Every Request Will Be Retried: Idempotency in Practice

A client sends a request. The connection drops before a response arrives.

For a separate people-operations application of the same measurement discipline, see online timesheets.

The client cannot determine whether the server processed it. A request lost on the way out and a response lost on the way back are indistinguishable from the client's position, and no amount of protocol design changes that — it is a property of communicating over an unreliable channel, not a deficiency in the implementation.

So the client has two options, and both are wrong some of the time: retry, and risk duplicate execution; or do not retry, and risk the operation never happening.

Duplicates are therefore structural. The question is not how to prevent retries but how to make the second execution harmless.

What idempotent actually means

An operation is idempotent if executing it more than once has the same effect as executing it once.

Naturally idempotent: setting a value (status = 'shipped'), deleting by identifier, upserting a row with a known key.

Not idempotent: incrementing a counter, appending to a list, creating a record with a generated identifier, sending an email, charging a card.

A common confusion: HTTP defines PUT and DELETE as idempotent and POST as not. That is a statement about what the method should mean, not a guarantee your handler provides. A PUT implemented as an append is not idempotent, whatever the specification says.

The idempotency key

The general mechanism, and the one payment APIs converged on.

The client generates a unique key per logical operation — not per HTTP attempt — and sends it with the request. The server records the key with the result. If the same key arrives again, the server returns the recorded result rather than executing again.

The important detail is in the ordering, and it is where most implementations are subtly broken.

Wrong:

1. Check whether key exists → not found
2. Execute the operation
3. Record the key and the result

Two concurrent requests with the same key both pass step 1 and both execute. The window is small and it is not zero, and under retry storms it is hit routinely.

Right — claim the key atomically first:

INSERT INTO idempotency_keys (key, state, created_at)
VALUES ($1, 'in_progress', now());
-- unique violation means someone else has it

The uniqueness constraint does the mutual exclusion. Whoever inserts successfully proceeds; whoever gets the violation either waits for the result or returns a "in progress" response.

Then the operation and the result must be recorded atomically together. If the operation commits and the key record does not, a retry re-executes. In a relational database, put both in the same transaction:

BEGIN;
  INSERT INTO idempotency_keys (key, state) VALUES ($1, 'in_progress');
  -- ... the actual work ...
  UPDATE idempotency_keys SET state = 'done', response = $2 WHERE key = $1;
COMMIT;

Where the work touches an external system that cannot join your transaction, you have a distributed commit problem and the transaction boundary does not save you. See below.

The details that decide whether it works

Key scope. A key must be unique per client and per operation type. A global namespace invites collisions between unrelated clients; a key reused for a different operation returns the wrong result.

Store the response, not just the fact. A retry should receive the same response body and status the original produced. Returning a bare 200 to a retry of a create tells the client nothing about which resource was created.

Handle the in-progress case explicitly. The second request may arrive while the first is still running. Options: block briefly and return the result, or return 409 and let the client retry. Both are defensible; silently executing again is not.

Record failures too, and decide which are replayable. A validation failure should return the same failure to a retry. A transient infrastructure failure should probably allow a genuine re-attempt. These are different, and treating them identically produces either stuck operations or duplicate work.

Set a retention period. Keys cannot be kept forever. Twenty-four hours is a common choice and it must exceed your longest retry window, or a late retry after expiry executes again.

Reject key reuse with different parameters. If the same key arrives with a different request body, that is a client bug and should be a 4xx rather than a silent return of the old result.

The unavoidable window

There is a case that idempotency keys do not close, and it is worth being explicit about rather than pretending otherwise.

If your operation must affect an external system — charge a card, send a message — you have two commits that cannot be made atomic. Either you record first and the external call fails, or you call first and the record fails.

The usual resolution is the outbox pattern. Write the intent to a table in the same transaction as the business change, and have a separate process read the table and perform the external call. The external call is then retried until it succeeds, and it must itself be idempotent — which is why payment providers require an idempotency key.

This gives at-least-once delivery, not exactly-once. Exactly-once across systems is not available. What the outbox provides is: the intent is durable, the effect happens at least once, and the receiver's idempotency makes repeated delivery harmless. See why exactly-once delivery does not exist.

Making operations idempotent when they are not

Client-generated identifiers. Have the client generate the primary key for a create. The insert then conflicts on retry instead of creating a second row. This is simpler than an idempotency key table and works whenever the client can generate the identifier.

Conditional updates. UPDATE ... WHERE version = $expected succeeds once and affects zero rows on retry. The zero-row result is information, not an error.

State machines with explicit transitions. UPDATE order SET status='shipped' WHERE id=$1 AND status='packed'. A retry matches nothing and changes nothing.

Deduplication on a natural key where one exists — an invoice number, an external reference.

Absorbing the increment. Instead of count = count + 1, record the event with an identifier and derive the count. Slower to read, correct under retry.

Testing it

Idempotency bugs do not appear in normal testing, because normal testing does not send the same request twice.

Send every request twice in your integration tests. Assert that the second returns the same result and that no additional side effect occurred. This one habit finds most of these bugs.

Send them concurrently, not sequentially. The sequential case usually works; the concurrent case is where the check-then-act window lives.

Test the interrupted case. Kill the process between the operation and the key record, and verify what a retry does. This is the hardest to arrange and the most likely to be broken.

The summary

You cannot avoid duplicates, because a lost request and a lost response look identical from the client.

So make the second execution harmless, either through natural idempotency or through a key claimed atomically before the work.

Store the response and handle the in-progress case, or you have moved the bug rather than fixed it.

And accept that exactly-once across system boundaries is not on offer — at-least-once delivery into an idempotent receiver is the achievable version, and it is enough.