I'm a software engineer at Datadog in New York. I previously worked at Bluecore , fixed interesting bugs at Twitter, and taught a database class at Columbia as an adjunct. I was a co-founder and CTO of Mitro, a password manager for groups and organizations , along with Vijay Pandurangan and Adam Hilss . Before that, I earned a Ph. D. from MIT , researching distributed OLTP databases with Sam Madden . Even earlier in my life, I worked at Google in New York for a bit more than a year, and I was a…
You can't safely use the C setenv() or unsetenv() functions in a program that uses threads. Those functions modify global state, and can cause other threads calling getenv() to crash. This also causes crashes in other languages that use those C standard library functions, such as Go's os.Setenv ( Go issue ) and Rust's std::env::set_var() ( Rust issue ). I ran into this in a Go program, because…
This is a reminder that random load balancing is unevenly distributed. If we distribute a set of items randomly across a set of servers (e.g. by hashing, or by randomly selecting a server), the average number of items on each server is num_items / num_servers . It is easy to assume each server has close to the same number of items. However, since we are selecting servers at random, they will have…
I was wondering: how often do nanosecond timestamps collide on modern systems? The answer is: very often, like 5% of all samples, when reading the clock on all 4 physical cores at the same time. As a result, I think it is unsafe to assume that a raw nanosecond timestamp is a unique identifier. I wrote a small test program to test this. I used Go, which records both the "absolute" time and the…
The read() and write() system calls take a variable-length byte array as an argument. As a simplified model, the time for the system call should be some constant "per-call" time, plus time directly proportional to the number of bytes in the array. That is, the time for each call should be time = (per_call_minimum_time) + (array_len) × (per_byte_time) . With this model, using a larger buffer should…
This is a post for myself, because I wasted a lot of time understanding this bug, and I want to be able to remember it in the future. I expect close to zero others to be interested. The C standard library function isspace() returns a non-zero value (true) for the six "standard" ASCII white-space characters ('\t', '\n', '\v', '\f', '\r', ' '), and any locale-specific characters. By default, a…
Nearly all programs are written to access virtual memory addresses, which the CPU must translate to physical addresses. These translations are usually fast because the mappings are cached in the CPU's Translation Lookaside Buffer (TLB) . Unfortunately, virtual memory on x86 has used a 4 kiB page size since the 386 was released in 1985, when computers had a bit less memory than they do today. Also…
Let's imagine we have an program that stores its state in a database, and we want other programs to do things when changes occur. For example, we might want to send email notifications if a bank balance drops below a threshold. This is a very common reason applications use message queues like Kafka. Unfortunately, the "trivial" implementation does not work when components fail. I suspect there are…
The Go "functional options" pattern is a way of passing options to a function. The function takes a variable number of arguments, which are themselves functions (a type like ...func(*config) . I think it was first introduced by Rob Pike in a 2014 blog post . It is now used by many APIs. For example, gRPC's DialContext() , AWS's LoadDefaultConfig() , and OpenTelemetry's Tracer.Start() . This style…
Following up on my last post about large JSON queries , I also benchmarked sub-string queries on large variable-length strings. I wanted to check if sub-string queries might be faster than HSTORE or JSONB key lookups. I tested both binary ( BYTEA ) and Unicode text ( TEXT ). Unfortunately, Postgres sub-string queries are about 6× slower than HSTORE or JSONB key queries. I also learned that…
Postgres supports three types for "schemaless" data: JSON (added in 9.2), JSONB (added in 9.4), and HSTORE (added in 8.2 as an extension). Unfortunately, the performance of queries of all three gets substantially slower (2-10×) for values larger than about 2 kiB, due to how Postgres stores long variable-length data ( TOAST ). The same performance cliff applies to any variable-length types, like…