There’s a really cool little mmap gadget that maps the same underlying memory region into two contiguous virtual memory address ranges. When you write past the end of the first mapped region, the remaining bytes end up in the second mapped region ( and at the start of the first one). Fabien Giesen calls this a “ Magic Ring Buffer ”, which is good enough for me. I wondered whether the virtual…
I posted recently about making kdb+ go faster . I forgot to mention shunning TCP and using shared memory as the transport layer. This has been around for ages and is the technology which underpins low-latency approaches like Pete Lawrey’s Chronicle Queue . You have choices about whether to make the file durable or in-memory only. We could simply use a shared library to write from kdb to such a…
Wherever I’ve worked on kdb+ installations the approach to authentication has been that there is no authentication. Generally, everyone’s just relied on the fact that you’d have to port-scan a host to find the kdb+ instances. To be fair, these have been extremely high-trust environments. However, if you have a requirement for strong authentication, you can use Kerberos not only for IPC but also…
I’ve recently been pondering how to improve the performance of kdb+. For what it’s worth, I think it’s going to be really hard to beat kdb+ at its “A game”, which is quant-analytics over big data. In a select template: we get free SIMD vectorisation there’s a simple time-and-space analysis tool it’s quick and easy* to write and amend [*] well … maybe not “easy” If we rewrote this in a language…
So … at “$Work” I’ve been doing quite a bit with our Dashboard product that is being used more and more to report on different aspects of the desk’s trading activities and KDB is a key part of delivering content. This of course has pushed us into the arms of Kx Systems’ c.js file (originally here ). This is another exercise in concission, but does come up short in a few areas, particularly…
I’ve written a new C library, which you can find in my Github KDB repository which does the same kind of thing as -11! , but allows you to provide a set of table names that will limit which messages are evaluated. For example if you had a market-data tickerplant and just wanted the trade messages (and not the more numerous quote messages), then this library could be of help. The library can be…
We have an application at work that upon restarting intraday needs to replay the market-data tickerplant log file and subscribe to the live data feed. This, of course, takes quite a while. Fortunately, we only subscribe to L1 equity data without futures. It’s a universe of around 6k names, give or take. If we want “real” data during development it’s all the more frustrating that we have to chew…
It used to be the case that we rewrote c.java to be more efficient. I haven’t downloaded it for ages, and have a feeling that it now uses primitive arrays under the covers for data-types which were formally stored as arrays-of-objects ( e.g. java.sql.Time[] ). Another improvement which is possible to make is processing as much of the IPC message as possible from whatever partial amount of input…
In my last post on a vectorised bitwise-OR function for kdb+, I wondered towards the end what the next interesting step would be — should it be the production of an AVX-variant of the same code, or should it be the modification of the code to handle short as well as byte values? Well, I opted for the former, and found out that it really wasn’t worth the trouble, and the biggest benefit was…
A long time ago I started writing some shared-library functions for kdb+ which would offer bitwise comparison of integer vector types using SSE vector registers. I came up with something which worked well enough but wanted to know if it could be made to go any faster. I wondered in particular about Intel’s prefetch instructions and needed a way of verifying whether they made any difference. Please…
This post is part of the series on performance monitoring with Intel MSRs on Linux: A Linux Module For Reading/Writing MSRs Intel MSR Performance Monitoring Basics Fun with MSRs: Counting Performance Events On Intel Scripting MSR Performance Tests With kdb+ Scripting MSR Performance Tests With kdb+: Part 2 Intel Performance Monitoring: Loose Ends: this post If you haven’t already, you’ll need to…
This post continues the series on performance monitoring with Intel MSRs on Linux using the batch-oriented kernel module to read and write values from and to the MSRs. The previous posts can be found here: A Linux Module For Reading/Writing MSRs Intel MSR Performance Monitoring Basics Fun with MSRs: Counting Performance Events On Intel Scripting MSR Performance Tests With kdb+ Scripting MSR…
This post is part of the series on performance monitoring with Intel MSRs on Linux: A Linux Module For Reading/Writing MSRs Intel MSR Performance Monitoring Basics Fun with MSRs: Counting Performance Events On Intel Scripting MSR Performance Tests With kdb+: this post Scripting MSR Performance Tests With kdb+: Part 2 Intel Performance Monitoring: Loose Ends One of the issues with coding…
This post is part of the series on performance monitoring with Intel MSRs on Linux: A Linux Module For Reading/Writing MSRs Intel MSR Performance Monitoring Basics Fun with MSRs: Counting Performance Events On Intel: this post Scripting MSR Performance Tests With kdb+ Scripting MSR Performance Tests With kdb+: Part 2 Intel Performance Monitoring: Loose Ends Hi, the last two posts have laid some…
This post is part of the series on performance monitoring with Intel MSRs on Linux: A Linux Module For Reading/Writing MSRs Intel MSR Performance Monitoring Basics: this post Fun with MSRs: Counting Performance Events On Intel Scripting MSR Performance Tests With kdb+ Scripting MSR Performance Tests With kdb+: Part 2 Intel Performance Monitoring: Loose Ends In the previous post I published code to…
This post is part of the series on performance monitoring with Intel MSRs on Linux: A Linux Module For Reading/Writing MSRs: this post Intel MSR Performance Monitoring Basics Fun with MSRs: Counting Performance Events On Intel Scripting MSR Performance Tests With kdb+ Scripting MSR Performance Tests With kdb+: Part 2 Intel Performance Monitoring: Loose Ends It’s been a while since the last post,…
Having just put the monster Relocations, Relocations blog-post to bed, at one point I caught myself trying to compute a relocation from the information given by readelf -r . It turns out that it’s a bit confusing, and not at all clear how you get from the readelf output to addresses and offsets. So, I’ve put together the following shared library in the hope that we can walk through that process.…
So, what’s a relocation? The man entry for elf contains the following: Relocation is the process of connecting symbolic references with symbolic definitions. While there are a few different types, the ones I’m interested in here are the runtime variety which enable the compilation of position independent code . In order to use absolute addressing, all the code – the application and all its DSOs –…
It’s not often you get a really nasty surprise when writing software, and even less often that the nasty surprise is lurking in the compiler (or assembler). It turns out that the Gnu assembler does not treat the operands of instructions such as fsub and fdiv uniformly — in fact, some of the time it does the opposite of what you instruct it to do. Were I to start ranting about this state of…
There’s no easy answer to that. I had applied to join a group on some social networking site which was for assembly language programmers, and to check that I hadn’t got them confused with a flat-pack furnishings fanciers, the polite email asked whether I would send them a simple encryption routine written in assembly which didn’t rely on xor . Since I seem to be playing with vector instructions, I…