Distributed systems, FoundationDB, Rust, and simulation testing, written from production experience.
The architecture I couldn't explain 🔗 One thing always puzzled me about FoundationDB: compared to many distributed databases, its architecture looks almost excessive: GRV proxies, commit proxies, resolvers, log servers, storage servers, and that's only the data plane. When I say that I operate FDB, people often ask, "Isn't that complicated? There are so many processes and roles."…
We have been using Apache Avro for years as one of the binary encodings for Materia 's records stored in FoundationDB. A few weeks ago, while writing the code that adds a new type of header for a new component, I made a mistake. I serialized the records, stored the bytes, and moved on. Only later did I realize I had forgotten something essential: I wasn't preserving the writer schema. At…
"I'm not going to write a PostgreSQL fake." My last post on why fakes beat mocks and Testcontainers kept coming up in conversations with colleagues, and that sentence was the most common reaction. They are picturing a reimplementation of a query planner, a write-ahead log, and twenty years of MVCC subtlety, and they are right to refuse that, because it was never the job. Fakes are way easier…
BugBash 2026 was two days in Washington D.C., organized by Antithesis, dedicated to extracting reliable software from the slop factory . The conference brought together thirty speakers from across the correctness landscape: Kyle Kingsbury running a seminar on transaction safety, Peter Alvaro reflecting on twenty years of attacking distributed systems, Brian Potter explaining why buildings…
Your CI pipeline spins up Kafka, Postgres, and Redis in Docker containers. It takes 4 minutes to start. Every test passes. And none of them can simulate the one failure mode that will page you at 3 AM. The problem is not speed or convenience. The problem is that Testcontainers give you a binary outcome: the whole service is up, or it is down. But production failures are partial . A Kafka broker…
Diving Into is a blogpost series where we are digging a specific part of the project's codebase. In this episode, we will dig into the implementation behind coverage-guided fuzzing. I have spent a lot of time building simulation tests . Sometimes I throw random operations at the system with different seeds, hoping something breaks. Sometimes I manually craft failure scenarios based on…
When you build a system on top of a key-value store like FoundationDB, you eventually need secondary indexes. You create them, you maintain them, and then one day you need to query them. Not just scan a single index, but combine results from multiple indexes: intersect them for AND conditions, union them for OR conditions, and fetch the actual records at the end. That's a query engine's…
People are right to fear LLM-generated code. The models hallucinate APIs, miss edge cases, and produce code that looks correct but fails under pressure. For distributed systems, where bugs hide behind race conditions and network partitions, the stakes are even higher. A subtle leader election bug can cause split-brain, data corruption, or cascading failures across your cluster. But what if you…
FoundationDB gives you serializable transactions with external consistency, automatic sharding, and fault tolerance. But once your first layer hits production under real load, you start seeing transaction conflicts you don't understand. The logic looks correct: read a key, check a condition, write the result. Under load, conflicts pile up and throughput collapses. How OCC Works 🔗…
In a few months, I went from skeptic to heavy user. Claude Code is now part of my daily workflow, both for my personal projects and at Clever Cloud where I help teams adopt these tools. I keep having the same conversation: colleagues ask how I use it, what works, what doesn't. This post captures what I tell them. The shift matters because code is becoming cheap . What used to take hours now…
2025 was the year I stopped managing and started shipping software again. After nearly two years of context-switching between fires and people issues, I returned to the keyboard. As the year closes, it feels like the right time to look back. It was about going deeper : into code, into writing, and into understanding why simulation testing is a superpower. Back in Engineering 🔗 The Transition 🔗…
I truly think LLMs are changing how we write software. For me, it's been a massive productivity boost. I can ask Claude to read some piece of code and explain it to me, or make a quick PoC of something, or refactor stuff that would take me hours. I even used it to help me backport features from FoundationDB in Rust , and it worked surprisingly well 🤯 João Alves made a great point recently :…
After one trillion CPU-hours of simulation testing , FoundationDB has been stress-tested under conditions far worse than any production environment. Network partitions, disk failures, Byzantine faults. FDB handles them all. But what about your code? Your layer sits on top of FDB. Your indexes, your transaction logic, your retry handling. How do you know it survives chaos? At Clever Cloud, we are…
Diving Into is a blogpost series where we dig into specific parts of a project's codebase. In this episode, we dig into Kubernetes' watch cache implementation. While debugging an etcd-shim on FoundationDB, I kept hitting "Timeout: Too large resource version" errors. The cache was stuck at revision 3044, but clients requested 3047. Three seconds later: timeout. This led me into the watch…
Diving Into is a blogpost series where we are digging a specific part of the project's codebase. In this episode, we will dig into the implementation behind FoundationDB's simulation framework. After years of on-call shifts running FoundationDB at Clever Cloud, here's what I've learned: I've never been woken up by FDB . Every production incident traced back to our code,…
While working on FoundationDB-rs , I hit a design problem that seemed like it would require complex trait gymnastics. I had two transaction types with identical APIs but different ownership semantics, and I needed functions to accept both. The solution turned out to be embarrassingly simple. It was already implemented. The Problem: Two Transaction Types, One API 🔗 FoundationDB-rs has two…
I keep having the same conversation with junior developers. They're building their first production service, and they ask: "Where should I put metrics in my application?" Then, inevitably: "What should I actually measure?" After mentoring dozens of engineers and running distributed systems for years, I've learned these aren't just beginner questions. Even experienced developers…
While working on moonpool , my hobby project for studying and backporting FoundationDB's low-level engineering concepts (actor model, deterministic simulation, network fault injection), Claude Code did something remarkable: it found a bug I didn't know existed on its own. Not through traditional testing, but through active exploration of edge cases I hadn't considered. Claude…
Summary: The difference between shipped and operated software is the difference between something you can run and forget, and something that demands ongoing, hands-on care. Choosing the former protects your team’s focus and sanity. The Shipped vs. Operated Spectrum 🔗 Some technologies arrive as complete systems: you deploy them, give them minimal care, and they quietly do their job. Others arrive…
I was listening to a couple of podcasts the other day and stumbled across two episodes that were so compelling I had to stop my chores and listen. They dive into corners of software engineering that most developers barely think about; not because they’re unimportant, but because they appear in the hard corners of engineering: catastrophic data corruption, correctness work done before a single line…
For years, I was a serial distro-hopper, working my way through Ubuntu, Arch, Gentoo, Exherbo, Void Linux, Fedora, Pop!_OS, and Manjaro. Every few months, a new Linux distribution would catch my eye, and I’d spend a weekend migrating my setup, hoping to find the perfect fit. That cycle broke three years ago when I switched to NixOS. It has since become the foundation for all my Linux machines, not…
That “YATTA!” Moment, Rebooted 🔗 We just merged at work our first successful data retrieval using DataFusion — a real SQL query, over real data, flowing through a system we built. And I’ll be honest: I haven’t had a “YATTA!” moment like this in years. This wasn't just a feature shipped; it felt like unlocking a new superpower for our entire system, a complex vision finally materializing. Not…
Introducing the FoundationDB Record Layer 🔗 Before we dive into the specifics of handling large operations with continuations (the main topic of this post), let's briefly introduce the FoundationDB Record Layer . It's a powerful open-source library built atop FoundationDB that brings a structured, record-oriented data model to FDB's highly scalable key-value store. Think of it as…
Tokio is the powerhouse of asynchronous Rust, celebrated for its blazing speed and robust concurrency primitives. Many of us interact with its core components daily— spawn , select! , async fn , and the rich ecosystem of I/O utilities. But beyond these well-trodden paths lie some incredibly potent, albeit less-publicized, features that can dramatically elevate your testing strategies, offer…
This article has been translated from my original French presentation at the upcoming Devoxx France 2025, titled " What if we embraced simulation-driven development? ". The Tale of a Bug 🔗 As a software engineer, my responsibilities include debugging distributed systems during on-call shifts. My tendency to attract peculiar issues during these shifts earned me the nickname "Black Cat". Let me…
I recently attended BugBash 2025 , a software reliability conference organized by Antithesis in Washington, D.C. during April 3-4, 2025. The conference brought together industry experts like Kyle Kingsbury, Ankush Desai, and Mitchell Hashimoto to discuss various aspects of building reliable software, with deterministic simulation testing being a significant focus throughout many of the sessions…
Ordered key-value stores like HBase, FoundationDB or RocksDB store keys in lexicographical order. When getting the latest version or most recent events, this ordering often requires scanning through all values in reverse order. While this works, it can become a performance bottleneck, especially in distributed systems. Let's explore a simple yet powerful optimization technique that I've…
FoundationDB is a powerful, distributed database designed to handle massive workloads with high consistency guarantees. At its core, the Data Distributor plays a critical role in determining how shards are distributed across the cluster to maintain load balance and resilience. In this post, we dive into the Data Distributor's internals, along with practical lessons we learned during a outage.…
As we approach 5 million downloads of the FoundationDB Rust crate (4,998,185 at the time of writing), I wanted to share some insights into how I ensure the safety of the crate. Being the primary maintainer of a database driver comes with responsibility, but I sleep well at night knowing that we have robust safety measures in place. Crate Overview 🔗 The Rust crate, foundationdb-rs , provides…
Time flies—it’s already 2025! Looking back, 2024 was an incredibly fast-paced year for me professionally as an Engineering Manager. This year, I’ve decided to take a new direction and return to a more engineering-focused role. I moved in a management position early 2023. It was a time where my company was growing fast (from 20-ish to 60-ish), and we needed coordination to ship things out in…
While FoundationDB allows you to obtain sub-milliseconds transactions’s latency without any knob-tuning, we had to bump a bit memory usage for Redwood under certain usage and workload. The following configuration has been tested on clusters from 7.1 to 7.3. BTree page cache 🔗 We discovered the issue when we saw a performance decrease on our cluster storing time-series data. Our cluster was…
I have been working around FoundationDB for several years now, and the new upcoming version is fixing one of the most evil and painful caveats you can deal with when writing layers: commit_unknown_result . Transactions with unknown results 🔗 When you start writing code with FDB, you may be under the assertions that given the database’s robustness, you will not experience some strange behavior…
I really like using RSS feeds. My Feedly account has more than 190 feeds, all neatly organized by categories. They help me keep up with new ideas and interesting blog posts about engineering. But there's another source of information I've been using for a long time that not many people know about: academic papers . You can discover details about infrastructure that you might not find in…
Learning distributed systems is tough. You need to go through a lot of academic papers, concepts, code review, before being able to have a global pictures. Thankfully, there is a lot of resources out there that can help you to get started. Here's a list of resources I used to learn distributed systems. I will keep this blogpost up-to-date with books, conferences, and so on. A distributed…
As I'm working on my latest contribution around FoundationDB and Rust , I had the chance to dig a bit into how FoundationDB's bindings are offering helpers to generate keys. Their approach is interesting enough to deserve a blogpost 😎 Row key? 🔗 When you are using a key/value store, the design of the row key is extremely important, as this will define how well: your scans will be…
Notes About is a blogpost serie you will find a lot of links, videos, quotes, podcasts to click on about a specific topic. Today we will discover ETCD. Overview of ETCD 🔗 As stated in the official documentation : etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It…
I’ve just realized that I’ve spent the last decade programming 🤯 While 2020 feels like a strange year, I thought it would be nice to write down a retrospective of the last 10 years 🗓 Learning to program 👨🏻💻 🔗 I wrote my first Hello, world program somewhere around September 2010, when I started my engineering school to do some electronics, but that C language got me. I spent 6 months…
TL;DR: I'm really happy to announce my latest open-source project called Record-Store 🚀 Please check it out on https://pierrez.github.io/record-store . What? 🔗 Record-Store is a layer running on top of FoundationDB . It provides abstractions to create, load and deletes customer-defined data called records , which are hold into a RecordSpace . We would like to have this kind…
Diving Into is a blogpost serie where we are digging a specific part of the project's basecode. In this episode, we will digg into the implementation behind ETCD's Linearizable reads. What is ETCD? 🔗 From the official website : etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or…
Notes About is a blogpost serie you will find a lot of links, videos, quotes, podcasts to click on about a specific topic. Today we will discover Raft's paper called 'In Search of an Understandable Consensus Algorithm'. As I'm digging into ETCD, I needed to refresh my memory about Raft. I started by reading the paper located here and I'm also playing with the amazing Raft…
This is a repost from OVHcloud's official blogpost. , please read it there to support my company. Thanks Horacio Gonzalez for the awesome drawings! This post has been published on both the StreamNative and OVHcloud blogs and was co-authored by Sijie Guo , Jia Zhai and Pierre Zemb . Thanks Horacio Gonzalez for the illustrations! We are excited to announce that StreamNative and OVHcloud are…
This is a repost from OVHcloud's official blogpost. , please read it there to support my company. Thanks Horacio Gonzalez for the awesome drawings! In today's blogpost, we're going to take a look at our upstream contribution to Apache HBase's stochastic load balancer, based on our experience of running HBase clusters to support OVHcloud's monitoring. The context 🔗 Have…
Notes About is a blogpost serie you will find a lot of links, videos, quotes, podcasts to click on about a specific topic. Today we will discover FoundationDB. Overview of FoundationDB 🔗 As stated in the official documentation : FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value…
Diving Into is a blogpost serie where we are digging a specific part of of the project's basecode. In this episode, we will digg into Kafka's protocol. The protocol reference 🔗 For the last few months, I worked a lot around Kafka's protocols, first by creating a fully async Kafka to Pulsar Proxy in Rust, and now by contributing directly to KoP (Kafka On Pulsar) . The full Kafka…
Diving Into is a blogpost serie where we are digging a specific part of of the project's basecode. In this episode, we will digg into the implementation behind Hbase's MemStore. tl;dr: Hbase is using the ConcurrentSkipListMap . What is the MemStore? 🔗 The memtable from the official BigTable paper is the equivalent of the MemStore in Hbase. As rows are sorted lexicographically in Hbase,…
In the last few months, there has been numerous blogposts about the end of the Hadoop-era. It is true that: Health of Hadoop-based companies are publicly bad Hadoop has a bad publicity with headlines like 'What does the death of Hadoop mean for big data?' Hadoop, as a distributed-system, is hard to operate, but can be essential for some type of workload . As Hadoop is based on GFS, we…
Among all features provided by HBase, there is one that is pretty handy to deal with your data's lifecyle: the fact that every cell version can have Time to Live or TTL. Let's dive into the feature! Time To Live (TTL) 🔗 Let's read the doc first! ColumnFamilies can set a TTL length in seconds, and HBase will automatically delete rows once the expiration time is reached . HBase Book:…
This is a repost from OVH's official blogpost. . Thanks Horacio Gonzalez for the awesome drawings! Handling OVH's alerts with Apache Flink 🔗 OVH relies extensively on metrics to effectively monitor its entire stack. Whether they are low-level or business centric, they allow teams to gain insight into how our services are operating on a daily basis. The need to store millions of…
Transaction? 🔗 "Programming should be about transforming data" --- Programming Elixir 1.3 by Dave Thomas As developers, we are interacting oftenly with data, whenever handling it from an API or a messaging consumer. To store it, we started to create softwares called relational database management system or RDBMS . Thanks to them, we, as developers, can develop applications pretty easily, without…
HBase? 🔗 Apache HBase™ is a type of "NoSQL" database. "NoSQL" is a general term meaning that the database isn’t an RDBMS which supports SQL as its primary access language. Technically speaking, HBase is really more a "Data Store" than "Data Base" because it lacks many of the features you find in an RDBMS, such as typed columns, secondary indexes, triggers, and advanced query languages, etc. --…