Principal software engineer with 20+ years building and operating distributed systems, developer platforms, and cloud infrastructure at scale. Writing about technology strategy, AI adoption decisions, and the engineering behind them.
The pipeline had degraded to the point of collapse. On the surface, the problem was that it converged on a choke point that was easy to saturate and hard to scale. But the real issue wasn’t having a bottleneck. It was pretending that there wasn’t one, treating back-pressure signals as a nuisance, and letting an AI-induced productivity anxiety run unchecked. The system, let’s call it The…
One of the hardest parts of integrating AI in our workflows is realizing that you can, and probably should, throw away assumptions and re-evaluate trade-offs from scratch. Unit tests are insufficient on their own to guarantee the correctness of a larger system so we rely on integration / e2e to fill the gap. But the latter are also slower and resource-hungry so as the number of tests grows,…
I’m no longer convinced that the classic code review process is the right tool for the job. For me, code reviews were always the prime resource for knowledge transfer, consistency, and collective ownership of code (not for finding bugs, that is what tests are for). I am still convinced that the goals remain relevant, perhaps more than ever. But I am not blind. In the teams I work with, every…
AI workloads break the “cattle” 1 approach to infrastructure management that made Kubernetes an effective IaaS platform. Kubernetes stays agnostic of the workloads, treats resources as fungible, and the entire stack underneath plays along: nodes on top of undifferentiated VMs on undifferentiated cloud infrastructure. It’s cattle all the way down. But AI infrastructure punishes mental models…
Was it ever possible to build a Proof of Concept that didn’t end up being rushed into production as a nest of bugs, instability, and technical debt, regardless of any advance warning from engineering that it will be “just a PoC”? Can we be at ease, now that anyone with a laptop has a PoC machine gun? Every PoC starts with good intentions. But the road to hell is paved with good intentions. They…
Everyone is focused on the impact of AI on the production of code. But code isn’t just produced, it has to be consumed: built, packaged, tested, distributed, deployed, operated. Leveraging AI to amplify the supply of code will grow already complex systems and accelerate the pace of change. Without a realistic plan to scale delivery pipelines, we’re asking for trouble. There is already smoke on the…
The median product engineer should reason about applications as composites of high-level, functional Lego blocks where technical low-level details are invisible. Serverless represents just about the ultimate abstraction for this mindset. Consider AWS Lambda ’s elevator pitch: “ you organize your code into Lambda functions [which run] only when needed and scale automatically. You only pay for the…
Once at Last.fm we had an integer overflow in an identify field. I can’t recall where exactly. But I do remember that the inconvenience of having a bunch of Hadoop jobs disrupted while we rushed to update the relevant type couldn’t spoil the collective pride for having more than 2 billion of whatever needed so many ids. Being frugal with identifiers is seldom a good idea, but for me the worst…
When you are bringing a new system to production you know that you ought to define SLIs, set up instrumentation, alerting, etc. Nowadays there is an abundance of tooling and infrastructure to extract data from your service and the entire stack it runs on. But this leaves you with a problem. What can we do with that data? Should you put all of it on a dashboard, in many? What should trigger an…
I have found that the concept of test types (unit, integration, and so on) does more harm than good. People often give me odd looks when I say this, so having the explanation in a URL will come handy for future reference. I will use it as an introduction to a series of practical case studies on the topic of testing software. Test types shape the way engineers and even adjacent professionals reason…