A few months ago I wrote about the new calculus of AI-based coding, how 10x velocity gains require corresponding investment in testing, deployment, and coordination, or the bottleneck just moves. I've been living that reality for a while now, and one piece of it has surprised me more than the others: the CI/CD pipeline. At ~100 commits a day, it stops behaving like a pipeline and starts behaving…
I've been doing agentic coding seriously for about six months now. In that time, I've developed some opinions about what works and what doesn't. I've also started to get conviction on what my ideal development environment would look like, a setup that doesn't fully exist today, but that I find myself wishing for constantly. The core insight I keep coming back to is this: the development…
I've been having a small problem. Not a big deal, but annoying enough that it kept bugging me. When working with AI agents on code changes, they sometimes struggle finding the right symbols. I'll give a prompt like "Let's change function cancel_job() to check the current state of the job, and return Error if the job is already completed", and the agent fires up shell scripts - grep commands, find…
When I blogged about the challenges and opportunities of AI assisted coding, there were a fair number of questions about details of our experience. I know many are eager to learn more, and we can't wait to share details of what we are building in a few weeks at re:Invent. However, among the questions, there was one that caught my attention: Looking at the git activity heatmap, you started coding…
Over the past three months, a team of experienced, like-minded engineers and I have been building something really cool within Amazon Bedrock. While I'm pretty excited about what we are building, there is another unique thing about our team - most of our code is written by AI agents such as Amazon Q or Kiro. Before you roll your eyes: no, we're not vibe coding. I don't believe that's the right way…
If you work with multi-tenant systems you are probably familiar with the concept of throttling or admission control. The idea is pretty simple and is rooted in the common human desire for fairness: when using a shared system or resource no customer should be able to consume "too much" of that resource and negatively impact other customers. What constitutes "too much" can vary a lot and will…
When working on distributed systems, it's not uncommon to need a piece of logic that executes across the entire system. Some examples of that can be a task that deletes tomb-stoned records, or a task that periodically updates system-wide configuration, or perhaps a piece of code that continuously scans the entire fleet replacing unhealthy hosts. In my experience, most distributed systems have at…
I was at an airport during a recent business trip, watching planes take off and land when I saw the iconic Boeing 747 take off. This reminded me how growing up, 747 was a favorite. There is just something special about a double-decker jumbo jet capable of flying long range routes. But the one factor that endeared me to 747 above all else were its four engines. Perhaps as a foreshadowing of my…
In this post, I wanted to make a quick observation about batching as an underused technique in distributed systems. Systems folks have long used batching as an effective way to increase the throughput of the system. This works by amortizing the overhead of an expensive action (such as device IO or a syscall) over multiple operations. Yet, outside of non-realtime systems, batching is rarely used…
The topic of software performance and efficiency has been making rounds this month, especially around engineers not being able to influence their leadership to invest in performance. For many engineers, performance work tends to be some of the most fun and satisfying engineering projects. If you are like me, you love seeing some metric or graph show a step function improvement - my last code…
In a recent operational review, a colleague brought up how past feedback from me has influenced his team's avoidance of feature flags. If we ever worked together, then you too probably heard me grumble about the downsides of feature flags. So should feature flags be avoided at all costs? My opinions on this topic are nuanced, so I thought it would be worth writing them down. But first, what do we…
Well, hello there. This is a sample post to see how things look.