Earlier this year, the program chairs for one of the largest machine-learning conferences found references in submitted papers to publications that did not exist. This was not a thought experiment about what a language model might do. It was not a benchmark in which researchers asked a model to write a literature review and then counted the invented citations. These were papers submitted to ICLR…
In this blog post, I discuss three days in July 2026 when a single agent session ran away from me, and what those three days revealed about the concurrency assumptions buried in our development tooling. Six weeks ago I wrote The Test Suite Was the Incident : my test suite had grown a pile of shared data nobody owned, every pull request paid to rebuild it, and the resulting failures had nothing to…
Toronto to Detroit to Norfolk. Two nights of Goose at the Dome, Monday and Tuesday, and I was flying in the night before. I have made this trip, or one shaped like it, more times than I can count. On Sunday night my connection to Norfolk slipped an hour, then another, then another. At 11:30 PM Delta cancelled it along with hundreds of other flights, said the next seat out was Tuesday, and told us…
“Thank god this app rocks and I can live through the chompers.” (a chomper, in the chat, on a night they could not make the show) Two months ago I wrote up Goose Spring ‘26 . The through-line of that post was that the live show finally worked. Live Activities stayed up, the setlist was seconds behind the room instead of minutes, song calls landed, and forty people spent a tour sending each other…
At 10:00 PM EDT last night I signed up for the $100 plan for Codex, the OpenAI coding agent that was driving most of the night’s work, because about a dozen Zabriskie PRs were in flight and I wanted them merged before I went to bed. Before 3:00 AM I had upgraded to the $200 plan. Somewhere in between, GitHub’s usage page ticked through roughly 90% of a $200 CI quota, and I sat there watching pull…
Someone on the Phish Facebook group reposted a TikTok overdub. Vanessa Bayer and Paul Rudd at a lunch table, losing their minds to a song while their coworkers stare. The original was Fleetwood Mac. Whoever made it swapped in “Down With Disease.” That move is Phish fans in miniature. Someone cared enough about the song and the bit that they rebuilt a piece of pop culture around the band. That’s…
I’m building Zabriskie , a social app for live music, mostly with a coding agent. I want to write something honest about what the last two weeks have actually felt like, because the data and the lived experience have been pointing at the same thing and I keep dressing it up in posts that argue for it more carefully than I need to. The honest version is: I feel like a goddamn babysitter. Every PR,…
I started this series because I’d been reading multi-agent papers for weeks and wanted a map I wished I’d had on day one. This is the last post. I want to close it by laying out what the field still hasn’t figured out, what I think is worth stealing from adjacent fields, and what I’d read if I had to start over. Getting Up to Speed on MAS Part 1. The Landscape Part 2. The Vocabulary Part 3. Wave…
If you’ve read this far, you’ve noticed that every paper I’ve discussed has a number next to it. 85.9 percent on HumanEval. 12.5 percent on SWE-bench. 25 percent on TravelPlanner. These numbers do a lot of work in the multi-agent literature, and they also do a surprising amount of harm. This post is about the benchmarks themselves. What they measure. What they don’t. And why ChatDev and MetaGPT…
Every agent system has to answer the same question eventually: how does it know it did the right thing? Wave-1 papers mostly don’t. Wave-2 papers get serious about it. Wave-3 papers measure what happens when they don’t. And the most interesting verification pattern in the field right now is one that isn’t in any paper at all. It’s in a commercial product. Getting Up to Speed on MAS Part 1. The…
If wave 1 was about role-playing and the agentic coding turn was about interface quality, there’s a parallel thread running through the field asking a more fundamental question: what should multiple agents actually do with each other? Debate? Share state? Coordinate? And are any of these interchangeable? This post is about four papers that sit at that intersection, including one that isn’t really…
By 2025, two things had happened. Wave-1 architectures were running in production (Anthropic had shipped its research system; the open-source ecosystem around orchestrator-worker patterns was maturing). The agentic coding turn had made clear that multi-agent was not the right tool for focused coding, and narrowed the interesting MAS question to “when we do use it, why does it break?” This wave is…
“this app is the bees. I am very grateful for it.” (a chomper, end of show, Irving, 4/25) Goose Spring ‘26 just wrapped. The run started March 28 in Athens and closed out Saturday night in Irving. Fourteen shows. One festival lead-in. Fourteen live chats running in parallel with each show, with 40 people sending 3,737 messages to each other across the tour. Most of them sent from couches. For the…
Wave 1 is the cluster of papers from 2023 that people actually cite. When someone says “I read the multi-agent papers,” they usually mean these. In this post I’m going to walk through the canonical five, explain what each one actually builds, and show where they agree and where they quietly disagree with each other. Getting Up to Speed on MAS Part 1. The Landscape Part 2. The Vocabulary Part 3.…
If you try to read multi-agent systems papers without the vocabulary, you will get nowhere. The field has settled on a shared set of words for the pieces of a system, and every paper now slots into those categories even when it pretends to be doing something novel. This post is about those words. Once you know them, you can read any paper in the field and know what it is and isn’t claiming.…
I’ve been reading multi-agent systems papers for weeks trying to figure out where the field actually is, and the honest answer is that it moves fast enough that any single paper is a snapshot, not a map. So this is the map I wish I’d had when I started. It’s a short series of posts meant to get someone up to speed on multi-agent LLM systems without having to read thirty papers first. Who this is…
By the fourth model swap I noticed that the part of the system I wasn’t swapping was the part holding the project together. Five configurations in three weeks: Opus 4.6, Opus 4.6 with the 1M context window, Cursor Cloud Agents on GPT Codex 5.3, Cloud Agents on Composer 2, and this week Opus 4.7. Some swaps were involuntary. Credits ran out. A model that had been my daily driver for two months…
I was halfway through another PR, regenerating the proof file for the fourth time after rebasing on main, when I realized I couldn’t name a single PR the permit gate had caught. Not “caught that other CI didn’t also catch.” Just caught . I decided to look up the numbers, and then I removed the gate entirely. What the Gate Was The short version: every PR to Zabriskie had to be accompanied by a…
Tonight, Goose was playing a very special set two. The band was spelling out SUCK IT STORM with their song titles, a response to a critic named Ryan Storm who’d publicly trashed their setlist choices and alternative arrangements, and people in the live chat were losing their minds about it in real time. Zabriskie was built for exactly this moment: friends sharing what they’re hearing, live chat…
I’ve been building Zabriskie for a few months now, mostly with AI agents. Claude Code writes the backend handlers, builds the SDUI screens, registers the routes, and, importantly, writes the Playwright tests. The test suite has grown to over 150 E2E tests. CI is green. I’m shipping fast. But I keep finding the same category of bug. The RSVP feature on show pages is a good example. A user taps the…
I’ve been using Cursor 3 more over the last week or so because it makes it easy to move between models, and Cursor’s Composer 2 has been producing good results for me. That’s become more important because Opus 4.6 has gotten noticeably worse for me in Zabriskie. Work that used to feel routine, like straightforward file edits or basic follow-through on multi-step changes, now takes forever across…
It’s 7 PM on a Thursday. I’m home after a day at work, watching Billy Strings on nugs as he plays the second night of a three-night run at the St. Augustine Amphitheatre. I switch over to the app I’ve been building, Zabriskie , a social music app for live shows, and expect tonight’s show page to just work. The show says “scheduled.” Billy Strings is literally on stage. People are in the venue. The…
I gave a guest lecture on AI in Michael Hilton’s Foundations of Software Engineering course ( CMU 17-313 ) today. One of my favorite things about lecturing is the conversations that happen afterward, the ones that go in directions nobody planned. This one hasn’t left my head: software engineering is going through the same transition that building went through in the 18th century, when structural…
I watched two Claude Code instances step on each other’s database migrations last month. One created migration 267. The other, running in a different worktree, also created migration 267. Different schemas, same filename. The second one silently overwrote the first. I stared at it for a minute before I started laughing. This is a lost update — the exact same problem that distributed databases have…
I’ve been building Zabriskie – a social music app for live shows – with Claude Code for about six weeks now. I’m building it because the jam band community deserves a real third place online, and the tools haven’t caught up to the culture. I also happen to be a distributed systems researcher, so I’m documenting what it’s actually like to build production software with an AI assistant – the wins…
“If I knew the way, I would take you home” — Grateful Dead, “Ripple” Claude has a persistent memory system. It can write lessons to files on disk and read them back at the start of every conversation. After each failure, it saves a note: don’t do that again. And then it does it again. The notes accumulate. The behavior doesn’t change. This is a story about one bug that happened five times, another…
“What a long strange trip it’s been” — Grateful Dead, “Truckin’” The Problem Nobody Talks About When I was doing my PhD in Europe — splitting time between Belgium, Portugal, and Paris — I got diagnosed with celiac disease. I’d gone to Europe partly because of the food. Paris. I wanted to teach there, live there, eat there. Then I found out I couldn’t eat bread, and in France that’s not a dietary…
“When life looks like Easy Street, there is danger at your door” — Grateful Dead, “Uncle John’s Band” (A note on picking this quote: I asked Claude to find me a Grateful Dead lyric that fit the theme. It couldn’t — searching for “dead lyrics” triggers the content filtering policy: API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"Output blocked by content filtering…
“Sometimes a notion gets a-hold of you, carries you away” — Grateful Dead, “Althea” The Saturday On March 7th, I sat down at 11am and started building. By 2am I had shipped a Relisten integration with inline audio players for archival recordings, a Quick Post feature for sharing past shows, Phantasy Tour as a live setlist source, Sign in with Apple, a badges and achievements system, security…
“Don’t let a heavy one hold back the dawn in you” — Goose, “(dawn)” The Third Place There’s a concept in sociology called the “third place” — not home, not work, but the place where you go to find your people. The coffee shop. The record store. The bar after the show. It’s where community happens. The internet used to be that place. In the late 90s and early 2000s, there were communities built…
Three weeks ago I wrote about building a social app in a week with Claude Code . The app shipped. My friends are using it. I kept building. Since that post, Claude has written 154 end-to-end tests across 17 spec files. It tests login, logout, signup, and redirect guards. It tests the feed, the bookmarks page, the notifications page. It tests liking, unliking, commenting, amplifying, recommending.…
Peter Van Roy and I just published a retrospective at PPDP ‘25 — the same venue where the original Lasp paper appeared a decade ago — looking back at ten years of influence from Lasp, the coordination-free programming model Peter and I built together in 2015. I wanted to write a bit about what that paper covers, and what it actually felt like to watch an idea travel from a prototype to something…
I spent the better part of a week building a social app with Anthropic’s Claude Code. Most of that work happened late at night, sometimes past 2am, iterating on features until I had something worth sharing with friends. Anthropic generates a weekly insights report for Claude Code users. Mine told an interesting story: 384 messages, 27 sessions, 168 files touched, six days. A median response time…
Ever want to test your system against Redis returning wrong values, like instead of returning an error, it returns an empty string? What about an empty byte array? What about a database field being null? You can do it with Filibuster 2.0! Let’s return a null from a Redis get, easy! Simply done with Filibuster.
Your developers are writing functional tests for their microservice and, when RPCs fail, they throw exceptions. Do they test the cases where it throws? Do they have tests for it? Using Filibuster, you can automatically identify these scenarios and prompt the developer to answer these questions. Here’s a case where a developer threw an exception when a downstream RPC failed. We ask them: “did you…
Ever wanted to just throw all sort of values at your database and see what happens to your application? Filibuster’s byzantine fault injector can take an arbitrary “value transformer” that looks like a functional fold, that allows you to come up with new fault injection scenarios as you inject faults! Let’s test our application against flipping characters in a string response from Redis! Here, we…
About two months ago I started prototyping this feature for visualizing API coverage in a microservice application through automated instrumentation: see where you have functional testing coverage, how many functional tests, and where you’re applying fault injection to determine the impact of those changes. Flash forward to yesterday, a new paper draft on arXiv proposed (almost) this very thing –…
You can also use Filibuster’s dynamic analysis linter to find microservice smells. Here’s one: invoking multiple RPCs to the same service because you can’t send them all together! This leaves you at risk for partial side-effects being applied: refactor your API to let developers supply all inputs and write data to your database transactionally!!!
You can also use Filibuster’s dynamic analysis linter to find microservice smells. Here’s one: using the arguments from one RPC to Service A as the inputs to a different RPC on the same Service A. You should refactor your API so I don’t have to make multiple RPCs!!
You can also use Filibuster’s dynamic analysis linter to find microservice smells. Here’s one: executing the same RPC multiple times, since you’ve already got the response and shouldn’t issue a failure-possible, expensive RPC again.
Integrated directly into IntelliJ, Filibuster can show you the RPCs that your service is making and directly inject faults on any of those RPCs. Attach a debugger, inject some faults, and see what your application does! Not only just GRPC (as shown here), we also support HTTP, Redis, PostgreSQL, CockroachDB, and DynamoDB.
Pushed out a new prototype Filibuster feature tonight: 🎉 use Filibuster’s IntelliJ plugin to get a “health check” of your microservice’s functional test suite. 🎉 Here, I can quickly open the plugin after running my test suite and see what RPC methods my service is exposing, how many unique functional tests I have covering those methods, how many specific Filibuster tests I have for those, and…
Ever wanted to test your microservice application against Redis failures? Filibuster 2.0 supports injecting faults against Redis, PostgreSQL, CockroachDB, and DynamoDB. Here, using our IntelliJ visualizer for tests, we see that in this test we injected a failure on a synchronous GET command to Redis and the test still passed. That’s fault tolerant code! Not using synchronous operations? No…
🎉 Happy to announce the 2.0 release of Filibuster (for JVM languages!) 🎉 This release includes: Lots of improvements for HTTP fault injection for Armeria’s HTTP clients and visualization of HTTP faults into the Filibuster IntelliJ plugin. Introduction of fault injection support for Redis, CockroachDB, DynamoDB, and PostgreSQL. Byzantine fault injection (value-based fault injection) support for…
What a long strange trip (the Ph.D.) it’s been: In 2018, when restarting my Ph.D. journey at Carnegie Mellon University, I wrote up a proposal for a Microsoft Research Fellowship that was sadly, rejected. In it, I said I wanted to make microservice programming more like monolithic programming with an integrated development environment. It even had fake screenshots we hacked with a fake LSP server.…
June 03, 2023 marks 10 years since I started my blog, and, it’s sort of mind blowing, honestly. When I go back and look at my old posts on my blog, well, it’s quite fitting that my first blog post was on fault injection and fault tolerance, because that’s the topic that ultimately became my Ph.D. research, of which I’ve been focused on since 2015… but really since 2018. Since creation of this…
I recently cleaned up Filibuster’s UX to display nested RPCs a bit clearer. Now, it will indicate using an arrow when RPCs are the result of other RPCs to dependent, transitive, downstream services.
Many microservice applications classify their dependencies into different Service Tiers . Perhaps your developers are working on a Tier 1 service and want to make sure that they are tolerant to any failure of Tier 2+ failures. How can they do that? Well, if they’ve written a functional test that can be run with Filibuster, you can use a new feature of Filibuster called fault injection filters .…
Testing microservice applications that are designed to be resilient, in order to find application bugs related to resilience, is difficult. This is different from testing microservice applications for resilience. Consider a microservice application using a custom RPC client that contains circuit breakers, and retry logic, and other resilience measures. Now, think about how we might want to test…
This addresses comments from LinkedIn on my previous blog post called Chaos Engineering or Software Testing? Last week, I took a bit of heat for my post on chaos engineering for couple reasons. First, it was said that I didn’t actually understand what chaos engineering was and thought the technique was purely random. Second, it was said that I failed to understand that chaos engineering has a long…