Practical engineering notes from Benjamin Cane, usually shared on LinkedIn first and kept readable on the open web.
“We can’t run locally” is usually a design smell. I’m a believer that, as an engineer, you should be able to run your software locally. But I hear it often: “We can’t run locally because of some reason.” Sometimes it’s valid. There are architectures and platforms out there that prevent running locally. But more often than not, when it comes to backend distributed systems, it’s a design or…
To make a service more stable, eliminate dependencies. One of the simplest reliability rules I’ve learned is this: Every dependency is another way for your service to fail. Why Dependencies Matter Every service has dependencies. Databases Caches Configuration services Secrets managers Logging pipelines Tracing backends All of these dependencies can fail, and when they do, the typical service will…
Caching isn’t hard. Some data is hard to cache. You’ve all heard the advice: “Avoid caching because caching is difficult to get right.” I agree with part of that statement. Caching can absolutely be difficult. But I think the reality is more nuanced. The difficulty level of caching depends heavily on the type of data you are caching. Not All Data Is Equal When engineers talk about caching…
The closer to the edge, the more stable a platform must be. The closer a component is to the customer, the greater its responsibility for keeping the entire platform available, even when everything behind it is having a bad day. Not All Services Carry the Same Reliability Burden Let’s consider a typical platform. Customer -> Load Balancer -> API Gateway -> Orchestrator -> Microservices -> Database…
Sometimes the most resilient thing a system can do isn’t retry. Most resiliency discussions focus on retries, timeouts, and circuit breakers. But some of the most important resiliency patterns happen after the failure. That’s where compensating transactions come in. Resiliency Is About Recovery A common mistake is thinking resiliency means preventing failures. Failures are inevitable. Networks…
Should retries and timeouts live in your application or your service mesh? This debate comes up constantly. Should resiliency live in the platform components, or should the application own it? Like most things in distributed systems, the answer is: It depends. Infrastructure Understands Traffic Service meshes, API gateways, and load balancers are great at handling generic resiliency concerns.…
Need to migrate from one database to another without downtime? Dual writes are one approach that deserves more attention. Most database migrations fall into one of a few buckets: Export and import Replication between two databases Services specifically built to synchronize data All of those approaches can work well. But sometimes you need both databases active while gradually migrating traffic…
Glue Services: Part Two — Data Synchronization. I recently talked about using glue services (Anti-Corruption Layers) to isolate modern platforms from legacy integrations. Today I want to talk about another type of glue service: data synchronization services. 🗃️ The Real Modernization Problem One of the hardest parts of replacing a legacy platform is usually not the application itself. It’s the…
When modernizing legacy systems, don’t be afraid to build glue services. One of the biggest mistakes I see during modernization efforts is letting legacy integrations dictate the design of the new platform. That usually leads to putting fresh paint on the same old house. Rebuilding the same architecture with a newer tech stack. 😴 The Dream vs. Reality The dream project is building a brand-new…
Coding agents can’t see your architecture diagrams—fix that. I’ve been talking a lot about architecture documentation and how it helps both humans and agents. From my experience, most teams spend 90% of their effort on diagrams and 10% on text. Why? Because they are the fastest way to communicate a system: Diagrams show how components interact, how data flows, and what depends on what. They help…
Most teams put low-level architecture in the wrong place—if they document it at all. I recently wrote about keeping architecture documentation in Git and making it available to coding agents. A common pattern I see is teams maintaining a central architecture repository that spans multiple services and platforms. It’s an approach I use as well—but there’s a nuance. 🏯 Architecture Has Layers Good…
Your coding agent is missing one thing: architectural context. I’ve been a big believer in Architecture Decision Records ( ADRs ) long before coding agents came along. Documenting decisions gives engineers context: Why is the system designed a certain way? What constraints existed at the time? What tradeoffs were made? That context matters. It also matters for agents. 🤖 Agents Need Context Too…
One of the easiest ways to break a gRPC service in production is health-checking the wrong listener. A common issue I see teams run into when adopting gRPC is leaving readiness checks pointed at their HTTP listener while production traffic actually flows through gRPC . Everything looks fine until it suddenly doesn’t. 🤔 The Problem Many gRPC services run two listeners: one for HTTP and one for…
Weighted load balancing has saved me more times than I can count. Many engineers think of load balancers as simple traffic distributors. Send requests across servers. Keep systems available. Move on. But one of their most valuable capabilities is often overlooked. Weighted load balancing. 🤨 What Is Weighted Load Balancing? From enterprise hardware appliances to software load balancers like…
YOLO is a terrible strategy for validating production changes. How many times have you seen it? Your platform is running smoothly. No alerts, no issues. Then suddenly, something breaks. After digging in, you discover the cause: another system you depend on made a change, and that change broke your platform. They didn’t notice it broke. You did, much too late… How many times have you been the cause…
Deterministic routing is one of the most effective ways distributed systems reduce consistency problems at scale. It is a foundational technique used by many modern databases, caches, and large-scale platforms. Understand how it works and you can apply the same pattern in your own systems. 🤔 Understanding the Problem At some point, every successful system hits the limits of a single database…
When you think of microservices, you probably think of centralized shared services. But there’s another valid pattern that is rarely discussed: running the same microservice inside multiple platforms. 🧩 How It Usually Works Most microservice designs follow the same model: Break systems into capabilities, teams, or functions Deploy one shared service for each capability Any platform that needs it…
Are you using traffic mirroring in production? If not, you might be missing one of the safest ways to test and observe production changes. 🚦 What is Traffic Mirroring? Traffic mirroring in Istio or Envoy Proxy lets you send a copy of live traffic to a secondary target. When enabled, traffic to /service routes to cluster1 as normal, and a mirrored copy is sent to cluster2 . The key: mirrored…
Use Agent Skills to capture institutional knowledge and make it usable by coding agents. Every organization has institutional knowledge. Internal frameworks Preferred practices Platform-specific capabilities It exists everywhere. But it’s often undocumented… or buried in a wiki no one reads. As coding agents take on more work, this problem gets worse. If you ask an agent to build a new service,…
Saved prompts are dead. Agent Skills are the next step. If you’ve been around for a while, you probably have a file full of bash one-liners. Small scripts or commands you saved because they solved a problem you didn’t want to automate properly. When coding agents arrived, prompts became the new one-liners. Useful prompts were saved, reused, and eventually turned into “prompt files”, then slash…
Generating code faster is only valuable if you can validate every change with confidence. Software engineering has never really been about writing code. Coding is often the easy part. Testing is harder, and many teams struggle with it. As tools make it easier to generate code quickly, that gap widens. If you can produce changes faster than you can validate them, you eventually create more code…
When you go to production with gRPC, make sure you’ve solved load distribution first. I was recently talking with another engineer who is rolling out gRPC into production. He asked what the biggest gotchas were. My first answer: Load Distribution. 🚦 HTTP/1 vs. HTTP/2 Most teams first implement services using REST over HTTP/1 and then migrate to gRPC as they seek its performance benefits. That…
You may be building for availability, but are you building for resiliency? Many teams design for availability. Far fewer design for resiliency. A concept that took me a while to really grasp is that building highly available systems and highly resilient systems is not the same thing. The difference is how the system reacts to failure. 🚄 High Availability When you build for high availability, the…
When your coding agent doesn’t understand your project, you’ll get junk. Junk in, junk out. One of the best ways to get more from agentic coding tools is to give the agent context. The more an agent understands your project, the better its work will be. If you ask an agent to add a method to a class, it will. It might read the file. It might infer some structure. But it won’t understand the…
You can have 100% Code Coverage and still have ticking time bombs in your code. 💣 I was listening to a team recently, and an engineer was discussing how a coding agent added additional tests to a project that already had 100% code coverage. The conversation reminded me that coverage is directional and often mistaken for quality. Just because your coverage shows 100% doesn’t mean your software is…