To make a service more stable, eliminate dependencies

Photo by Shubham Dhage on Unsplash.

To make a service more stable, eliminate dependencies.

One of the simplest reliability rules I’ve learned is this: Every dependency is another way for your service to fail.

Every service has dependencies.

• Databases

• Caches

• Configuration services

• Secrets managers

• Logging pipelines

• Tracing backends

All of these dependencies can fail, and when they do, the typical service will fail with them.

The more dependencies a service has, the more failures it inherits. Every dependency adds features, but it also adds failure modes.

Reliability is often about deciding which dependencies are actually worth the tradeoff.

I recently wrote about how systems closest to the customer carry the greatest responsibility for availability.

This is one reason edge systems tend to be dependency-light.

Load balancers, API gateways, and routers are responsible for availability.

Every dependency added to these systems creates another opportunity to take down the entire platform. So they tend to avoid dependencies whenever possible.

Sometimes removing a dependency entirely isn’t practical.

A better question is:

• What happens if the dependency disappears?

• Can the service continue operating?

• Can it use cached data?

• Can it fall back to a last-known-good configuration?

• Can it degrade gracefully?

If the answer is yes, you’ve significantly improved reliability even though the dependency still exists.

Take Envoy Proxy. Envoy can receive configuration from an xDS service. But it doesn’t call xDS for every request.

Instead:

• Configuration is fetched periodically

• Stored in memory

• Used locally during request processing

If the xDS service becomes unavailable, Envoy continues routing traffic using its last known configuration. The dependency still exists, but request processing no longer depends on its availability.

That’s a very different reliability model.

One of the most common mistakes I see is making observability a required dependency.

If your logging or tracing backend becomes unavailable, should customer traffic stop flowing? No.

In most cases, observability should be a best effort.

Use asynchronous logging, buffering, and truncation policies so customer traffic continues to flow even when your observability platform is experiencing issues.

Operational visibility is important, but customer availability is more important.

You’ll never eliminate every dependency. But you can eliminate unnecessary ones.

And for the remaining dependencies, you can design your service to survive failures.

The most reliable services aren’t dependency-free. They’re designed to survive dependency failures.

Originally posted on #Bengineering.

If this hit home, share the original with someone who needs it: To make a service more stable, eliminate dependencies

• Caching isn’t hard. Some data is hard to cache
  Dependency reduction often starts by being selective about which remote data you truly need on demand.

• The closer to the edge, the more stable a platform must be
  Fewer dependencies usually matter most in the parts of the platform closest to users and traffic spikes.

• Should retries and timeouts live in your application or your service mesh?
  It complements this post by showing where dependency-handling logic should actually live.

• Most teams put low-level architecture in the wrong place
  Dependency choices are easier to improve when the component-level design is documented clearly.

• In-Flight Request Tracking: Lessons from Card Payments and HTTP/2
  A deeper look at uncertainty, timeouts, and correctness when distributed requests do not complete cleanly.

• Are Atomic Operations Faster and Better Than a Mutex? It Depends
  A practical reminder that simpler-looking performance choices still need correctness tradeoff analysis.