Kun Chen, whom I interviewed last week, has a fun post up where he had various AI approaches mimic the funny “tech company org chart” meme that everyone loves. As with many things, it’s funny because it’s true:
Kun organized a group of agents based on the various org charts and told them to build a spreadsheet. Here are the results:
Apple (one leader, eight workers, no worker-to-worker contact): The app looked beautiful, but typing =SUM(A1:A3) displayed the literal text instead of calculating because the formula engine was never connected to the rendering layer. Oops!
Amazon (three layers of management): The layers played Chinese whispers with each other, ignored the written spec, and finished quickly… but a bunch of stuff did not work.
Facebook (everyone can talk to everyone): Moved really fast but shipped a spreadsheet where =SUM(10, 20, 30) returned 40 instead of 60.
Microsoft (two rival divisions competing): Both divisions piled onto the same visible features (clipboard handling) and ignored less glamorous required features. Lots of duplication. Shipped with a bunch of stuff that did not work.
Oracle (one leader, four specialist reviewers): Took forever, passed compliance, but the product did not work.
Google (two layers, design docs required): The winner. Best app. Reasonable cost.
The lesson is not that Google’s approach is the best. The lesson is that no topology is perfect and every structure necessitates blind spots you can predict in advance. Apple had a single bottleneck, so anything requiring coordination between subsystems was going to suffer. Amazon ignored the spec, so the design was going to drift. Facebook had no one responsible for the final product, so no one checked it. Etc.
Real life org structures are as much (more!) about human motivations in general and specific relationships in particular, but I think we can glean some architectural insights from these network topologies as we figure out how to build and operate code factories.
Design docs catch what’s in them, not what’s missing from them. By definition, a system cannot judge itself from the inside. You must have an external source of validation, ideally from a different model or a human. None of the topologies caught their own product failure (in retrospect, this obviously must be true).
Make a canonical spec and route traffic through it. The spec earns its keep when every PR has to cite it, especially for functional requirements.
Wherever two components have to fit together, the agents responsible for them need to be able to talk directly. Same thing happens in human organizations, and this is why they tend to ship their org charts.
Any code factory process, therefore, ought to invest a great deal of time getting the spec right upfront, and use a workflow where the agents must refer to the spec at each step.
This will make sure what gets built is what was described, but it may not be what was actually desired. So you need an external validator to see whether “to spec” equals “correct” at the product level.
At the seams, where two systems connect, the agents responsible must speak with each other directly.
The integrator, the role that determines what gets shipped and what does not, is the most consequential design choice. It drives how fast work moves, who gets reviewed, what is missed, etc. “how does code get integrated” is the first question, not the last. Most other roles can be filled in around the integrator structure once it’s chosen.
Two questions matter: how many integrators, and what do they share. One integrator means a serial bottleneck. Many integrators with no shared reference means inconsistent decisions. Many integrators with a shared reference (a spec, a design doc, a contract) mean clean parallelization.
There is no “best” solution, just tradeoffs, but if I was not aggressively optimizing for efficiency or correctness, and also not 100% sure if my design was correct, I would:
Require a well-specified design-doc for the parts of the work that are well understood, with multiple integrators who must rely on this doc.
Have a separate exploratory phase upstream of the spec where the design problem itself is being figured out
Have an aesthetic evaluation role outside of the development work
Have an external, independent evaluation at the end
If anyone has any other thoughts re: agent topologies and code factories I would love to learn!
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.