This is a blog on distributed systems. I am a systems researcher currently working at Meta. Previously I worked as a software engineer at Confluent (2022-2024) and Facebook (2017-2022); an Associate Professor at Yale; and a researcher at VMware Research and Microsoft Research Silicon Valley. All opinions are my own.
I’ve been building systems, teaching system design, and writing papers about novel designs for a couple of decades now. In my view, agents are magical self-writing distributed programs: my personal research agenda has swivelled to designing abstractions that can make them safe when they interact with complicated environments. But as I design and write1 this new system (LogAct / AgentBus), I have…
We start with a re-definition and an observation.
“It used to be said there were two kinds of chairs to go with two kinds of Ministers: one sort that folds up instantly, the other sort goes round and round in circles.” — Bernard Woolley, Yes Minister [when the minister asks for a new chair]
Since moving from academic research to industry in 2017, I’ve worked on two software projects. Each one started as a small, clean-slate1 skunkworks effort involving 2-3 people and gradually expanded to a large, conventional software engineering effort with dozens of engineers. The first of these (from 2017 to 2021) was Delos at Meta, a Chubby/ZooKeeper/etcd-like control plane storage system. The…
I posit that most software engineers (particularly those working on infrastructural systems) are destined to wallow in unnecessary complexity due to three fundamental laws.
Early in my research career, I had a chance to work with some of the best system researchers1 in the world on a number of really interesting system designs. One of the enjoyable aspects of research was the particular process used by researchers (particularly in the SOSP/OSDI community) to come up with novel yet practical designs. This design process can be characterized as “fighting complexity…
I’ve heard multiple times that a strong notion of leadership somehow simplifies replication. I don’t think this is true. I explain why in this post.
There are at least five distinct paradigms for replication: Group Communication [0], Viewstamped Replication [1], MultiPaxos [2], Raft [3], and Shared Logs [4]. In a previous post, I did a deep-dive on MultiPaxos, showing that it implements a specific abstraction: State Machine Replication or SMR. The SMR API allows servers to propose commands and play them back in a durable total order:
There are three questions to ask of any system: What abstraction does it implement? What is the design space for such an abstraction? Why is this abstraction useful? In a previous post, we examined the Paxos protocol and answered the first two questions. Paxos implements the abstraction of a Write-once Register (a WOR) using a combination of quorums and a two-phase locking protocol. As for the…
The road to Paxos is a long one (as with other greek islands) and also somewhat elusive (it’s an island, after all). It took me longer than I’d like to admit to obtain a working understanding of the Paxos protocol. In my early attempts, I’d hit a brick wall of complexity: do I really need to know what this particular acceptor is going to do? What’s a learner anyway? What does it even mean to…