Sometimes the most resilient thing a system can do isn’t retry

Photo by Clay Banks on Unsplash.

Sometimes the most resilient thing a system can do isn’t retry.

Most resiliency discussions focus on retries, timeouts, and circuit breakers.

But some of the most important resiliency patterns happen after the failure.

That’s where compensating transactions come in.

A common mistake is thinking resiliency means preventing failures.

Failures are inevitable.

Networks fail. Services crash. Messages get lost. Requests time out.

True resiliency is accepting that failures will occur and having a plan to recover when they do.

Sometimes recovery means correcting actions that may already have occurred.

Have you ever swiped your card at a store, received a charge notification, only to have the terminal report an error and later discover that the charge was nowhere to be found?

What happened?

The point-of-sale terminal sent an authorization request.

But before it received a response, something failed: a network issue, a timeout, or a problem somewhere in the payment flow.

Now the terminal doesn’t know whether the transaction succeeded or failed.

The authorization request may have reached the card issuer and been processed, or it may not have.

Rather than risk leaving the customer incorrectly charged, the terminal sends a second transaction: a reversal.

That reversal is a real-world example of a compensating transaction.

Its purpose is simple:

Undo the effects of a previous action if that action completed successfully. If the original authorization never happened, nothing changes.

If it did happen, the reversal corrects it. Instead of determining exactly what happened, the system performs a corrective action.

Compensating transactions show up in many real-world distributed systems:

• Releasing inventory after a failed order

• Refunding a payment after a fulfillment failure

• Canceling a reservation when verification times out

The pattern is always the same: something failed, and the system takes a corrective action.

When engineers think about resiliency, they often focus on preventing failures. But distributed systems fail in unexpected ways.

Sometimes the most resilient thing a system can do isn’t retry. It’s correct the mistake and move forward.

Originally posted on #Bengineering.

If this hit home, share the original with someone who needs it: Sometimes the most resilient thing a system can do isn’t retry

• Glue Services: Part Two — Data Synchronization
  A useful companion on keeping old and new systems aligned during gradual platform change.

• When modernizing legacy systems, don’t be afraid to build glue services
  Edge translation and isolation patterns pair well with migration-focused architecture decisions.

• YOLO Is a Terrible Strategy for Validating Production Changes
  Operational changes get safer when validation and rollout discipline are part of the design.

• Coding agents can’t see your architecture diagrams—fix that
  Systems thinking improves when design context stays readable, versioned, and close to the code.

• In-Flight Request Tracking: Lessons from Card Payments and HTTP/2
  A deeper look at uncertainty, timeouts, and correctness when distributed requests do not complete cleanly.

• Are Atomic Operations Faster and Better Than a Mutex? It Depends
  A practical reminder that simpler-looking performance choices still need correctness tradeoff analysis.