Third time's the charm: building a low-latency realtime database

When I set out to build Lark -- a Firebase Realtime Database alternative -- I had one non-negotiable requirement: predictable, low latency. My goal was to keep internal p99 latency at less than 50ms, and p50 latency at less than 10ms. That single requirement led me through three distinct architectures before I found one that worked, and taught me several lessons that might prove useful to others…