Lessons from Scale #10: Failure Is a First-Class API

Failure is not an edge case.

At scale, systems fail constantly:

• dependencies time out
  
• partial work succeeds
  
• retries collide
  
• state drifts
  
• networks lie

The real problem isn’t that things break.
It’s that most systems pretend failure is exceptional — and act surprised when it happens.

That pretense is where humans sneak back in.

Where humans appear

Across this series, humans show up in the same places every time:

• when systems can’t decide whether to retry
  
• when partial success needs interpretation
  
• when “did this already run?” isn’t knowable
  
• when the blast radius of a mistake is unclear

People aren’t fixing bugs.
They’re compensating for undefined failure semantics.

If failure isn’t modeled, named, and exposed, users will invent their own handling.
And guessing is the most expensive failure mode.

Failure is part of the interface

Most APIs describe how things succeed.

Very few describe how things fail.

Users need to know:

• what failed
  
• what didn’t fail
  
• whether it’s safe to retry
  
• whether the system will retry
  
• what guarantees still hold

Without that information, users do the only thing they can:
they ask a person.

That’s not resilience.
That’s delegation.

Idempotency is a promise

Idempotency is often treated as an implementation detail.

It isn’t.

It’s a contract.

When a system claims an operation is safe to retry, it’s making a promise to users about side effects, duplication, and state consistency. AWS is explicit about this framing in its documentation on idempotent APIs: the goal isn’t convenience — it’s preventing accidental damage when failure inevitably occurs.

If that promise isn’t explicit, users won’t trust retries — and they shouldn’t.

Retries without semantics are denial

Retries are not a failure strategy.

They’re an admission that failure happened — followed by hope.

Without clear semantics, retries:

• amplify load
  
• duplicate work
  
• corrupt state
  
• hide real problems

Google’s Site Reliability Engineering guidance is blunt about this. In Handling Overload, reliability comes from understanding failure modes, not from assuming retries will save you.

Retries only work when failure is understood, bounded, and intentionally handled.

Compensation is how workflows stay honest

In real systems, not everything can be rolled back.

That’s where compensation comes in — not as an afterthought, but as part of the design.

Modern workflow engines treat compensation as a first-class concept because they assume:

• partial success is normal
  
• failure is expected
  
• progress must be durable

Temporal’s workflow model is explicit about this: failures are recorded, compensation is deliberate, and progress is durable.

That’s not pessimism.
That’s realism.

The series, closed

This series started with a simple observation:

Humans don’t scale.

It ends with the reason why they keep reappearing.

• Workflows replace humans as the integration layer
  
• Observability replaces humans as interpreters of state
  
• Failure semantics replace humans as decision-makers under stress

A system that handles failure well feels calm.
A system that doesn’t feels fragile — even when it works.

The lesson

At scale, success is the easy path.

Failure is where systems reveal what they actually are.

If failure isn’t a first-class API, humans will fill in the gaps.

And humans don’t scale.