How Retries and Idempotency Actually Work

Retries sound simple until money moves, stock decrements, jobs fan out, or a payment rail answers slowly enough to make the caller doubt reality.

That doubt is the whole problem. A retry is not really about trying again. It is about what the caller should do when it cannot tell whether the first attempt took effect. A TCP connection resets after the request body was uploaded. A reverse proxy returns 504 Gateway Timeout while the upstream worker keeps running. A mobile app loses radio coverage after tapping pay. A job worker crashes after charging a card but before persisting the success row. The caller sees uncertainty, not a neat success or failure.

If the system responds to that uncertainty with blind repetition, it creates duplicate side effects. Two charges. Two outbound emails with different links. Two shipments. Two ledger postings. If the system refuses every retry to avoid duplicates, it turns ordinary packet loss and transient overload into visible product failure.

This is why retries and idempotency belong together. Retries are the policy for dealing with uncertain completion. Idempotency is the mechanism that makes repeated attempts converge on one logical outcome.

This article explains the mechanics without slogans. We will look at why timeouts are ambiguous, what HTTP does and does not guarantee, how to place the duplicate guard around the real side effect, what an idempotency record needs to store, how queues and downstream systems extend the problem, and why backoff, deadlines, and observability matter as much as the key itself.

A Retry Is a Response to Ambiguity, Not a Vote of Confidence

Most production retries do not happen because the caller knows the first attempt failed. They happen because the caller does not know.

Suppose a shop platform in Barcelona calls a payment API in Amsterdam to authorise €48.00 for an order. The client sends:

POST /payments HTTP/1.1
Host: api.payments.example
Content-Type: application/json
Idempotency-Key: ord_2026_05_30_4819_attempt_1
 
{"orderId":"ord_4819","amount":4800,"currency":"EUR"}

Several bad things can happen after that point.

1. The request never reaches the API at all.
2. The API receives it, validates it, and rejects it with a clear 400.
3. The API starts processing, calls a downstream rail, commits the result, and then the response is lost on the way back.
4. The API calls the rail successfully, crashes before storing completion, and the client times out.
5. A proxy in front of the API gives up first and returns 504, even though the upstream worker will finish ten seconds later.

From the caller's point of view, cases 3, 4, and 5 can all look like the same thing: timeout.

That is why timeout handling is dangerous. A timeout is not a domain outcome. It is a transport observation made by one participant. It says, "I did not receive a complete answer before my deadline." It does not say, "the operation definitely did not happen."

A useful mental split is:

• definitive failure: the system can prove the operation did not take effect
• definitive success: the system can prove the operation took effect
• ambiguous outcome: the observer cannot tell

Retries exist for the third case.

This sounds abstract until you look at logs. Real incidents often look like this:

12:03:11.204 client  -> POST /payments key=ord_4819
12:03:11.241 api     -> reserve idempotency key
12:03:11.417 rail    -> auth approved auth_id=au_90812
12:03:11.430 api     -> commit payment row id=pay_712
12:03:14.205 client  -> timeout waiting for response
12:03:15.009 client  -> retry POST /payments key=ord_4819

If the second request is treated as a fresh payment, the system is wrong. If it is tied back to the first logical operation, the system can replay the known result and stay correct.

The important point is that retries are not only a network concern. They are business correctness under uncertainty.

HTTP Defines Some Idempotent Method Semantics, but Your Handler Still Owns the Effect

HTTP already has the word idempotent. RFC 9110 defines methods such as GET, HEAD, PUT, and DELETE as idempotent in intent. Repeating the same request should have the same intended effect on server state as doing it once.

That matters, but it is not enough.

First, the RFC talks about semantics at the interface level, not about every side effect a real handler may trigger. A PUT /profile/42 that writes the same profile document twice is idempotent in resource semantics. The same handler may still emit two analytics events, two cache invalidations, and two emails if the implementation is sloppy.

Second, POST is not idempotent by default, yet many important business operations need to be retried safely and therefore need idempotency anyway. Creating a payment, placing an order, issuing a refund, or provisioning a server is often modelled as POST. The absence of built in idempotent semantics does not remove the operational need.

Third, even methods that are idempotent in theory can stop being so if the server defines them carelessly.

Examples:

• DELETE /sessions/abc can be idempotent if repeated deletes keep returning the same conceptual outcome such as "session absent".
• DELETE /invoice/4819 is not magically safe if the first delete triggers irreversible archival and the second delete trips a compensation bug.
• PUT /stock/sku123 can be idempotent if it means "set stock to 8".
• POST /stock/sku123/decrement is not idempotent if it means "subtract 1 each time received".

What matters is the business effect.

A helpful distinction is:

• safe: the operation does not change server state in the first place
• idempotent: the operation may change state, but repeating the same logical request does not change the final effect beyond the first successful application

A GET is normally safe and idempotent. A PUT is normally not safe, but can be idempotent. A payment capture is definitely not safe, but it still often needs idempotent retry behaviour.

That is why mature APIs explicitly document retry semantics per endpoint rather than relying on method names alone.

Bad documentation says:

POST /payments
Creates a payment.

Useful documentation says:

POST /payments
Accepts an Idempotency-Key header.
The same key with the same request body returns the original result.
The same key with different parameters returns 409 Conflict.
Keys are retained for 24 hours.
Clients may retry on network timeout, connection reset, 429, and 503 using exponential backoff with jitter.

The method name sets a baseline. The application contract decides whether real retries are safe.

The Hard Part Is the Failure Window Around the Side Effect

Every retry design has to start with the same question: where is the side effect, and what can fail before and after it?

Take a payment authorisation path:

client request arrives
-> validate body
-> reserve duplicate guard
-> call payment rail
-> persist payment state locally
-> send HTTP response

There are four important windows.

1. Failure before the side effect starts

Validation fails. Authentication fails. The request is rejected before anything external or durable happens.

This is the easy case. The caller can retry only if the error class makes sense to retry. There is no duplicate side effect risk because nothing happened.

2. Failure before local completion is durable

The server started work but did not yet persist a result. Perhaps it called an internal service, then crashed before updating its own database. Perhaps it allocated a worker slot and timed out before the downstream rail answered.

This case is dangerous because the system itself may not know whether the effect happened elsewhere.

3. Failure after the side effect committed but before the caller learned it

This is the classic idempotency case. The charge succeeded. The row committed. The response packet never made it back.

The caller sees timeout. The system must recognise the retry as the same logical operation and return the stored result instead of doing the work again.

4. Failure after local completion but before all downstream consequences finished

The API returned 200, but the outbox dispatcher, email sender, warehouse reservation, or webhook delivery still has to happen. The primary effect may be complete while secondary effects are still pending.

If the system models those later steps badly, retries or replays can still create inconsistency even though the main HTTP request looked correct.

This is the reason "exactly once" claims should make you suspicious. In distributed systems, the real guarantee is usually narrower and more practical:

1. identify one logical operation
2. make the primary state transition happen at most once
3. make later repeats return or converge on that same outcome
4. design secondary consumers so they deduplicate too

Anything stronger needs very careful qualification.

A simple timeline makes the issue concrete:

T1 client sends POST /payments key=K
T2 server reserves K
T3 rail authorises payment
T4 server commits payment row
T5 response lost
T6 client retries POST /payments key=K

At T6, the duplicate guard has one job: map the second request back to the already completed operation at T4.

If the guard sits only after the rail call, it is too late. If it is non durable, a crash erases it. If it ignores request parameters, it can accidentally merge two different operations. The placement and content of the guard determine whether retries are a safety feature or a duplicate generator.

Idempotency Works by Naming One Logical Operation and Reusing That Name

The simplest useful definition of an idempotency key is: a caller supplied name for one logical operation.

That name needs some discipline.

It must identify the business attempt, not the TCP attempt

If the mobile app times out and retries the same order placement three times, those three transport attempts should all carry the same idempotency key.

If the app generates a fresh key for every retry, the server cannot correlate them.

It should be scoped

A good system normally treats a key as unique within a namespace such as tenant plus endpoint class. create-payment:ord_4819 and create-refund:ord_4819 should not collide just because the raw suffix is the same.

It must be bound to the request parameters

Otherwise the caller can accidentally or maliciously reuse the same key with a different payload and get a nonsense replay.

For example:

POST /payments
Idempotency-Key: ord_4819
 
{"orderId":"ord_4819","amount":4800}

Later:

POST /payments
Idempotency-Key: ord_4819
 
{"orderId":"ord_4819","amount":5800}

Those are not the same logical request. A robust implementation stores a canonical request hash alongside the key and rejects mismatches.

It needs a retention window

Keys cannot live forever in the hot duplicate store unless you want unbounded growth. Systems therefore retain them for some defined period such as 24 hours, 7 days, or a business specific window.

That retention period is not a mere storage choice. It is part of the correctness contract. If clients may retry for 48 hours because mobile devices roam in and out of coverage, but the server evicts keys after 6 hours, the guarantee is weaker than the product team probably thinks.

A realistic request might look like this:

POST /payments HTTP/1.1
Host: api.payments.example
Content-Type: application/json
Idempotency-Key: 9a8a4c11-0bc5-4d9b-8b2c-780ef4f4f7f2
 
{
  "merchantId": "m_144",
  "orderId": "ord_4819",
  "amount": 4800,
  "currency": "EUR",
  "capture": true
}

The server should conceptually interpret that as:

If key K for merchant M has never been seen with this parameter hash, execute once.
If key K for merchant M already completed with the same parameter hash, return the stored result.
If key K for merchant M exists with a different parameter hash, reject.

That is the core algorithm. Everything else is engineering detail around making it durable, concurrent, and observable.

The Server Needs a Durable Request Journal, Not Just a Fast Cache

Many weak implementations treat idempotency as a response cache keyed by a header string. That can work for demos. It is not enough for important writes.

A serious implementation usually needs a durable request journal with at least:

• tenant or caller scope
• idempotency key
• canonical request hash
• status such as started, completed, or failed
• response code and response body for replay
• timestamps for creation and expiry
• optionally a pointer to the domain object created

A PostgreSQL sketch:

CREATE TABLE idempotency_requests (
  tenant_id           text        NOT NULL,
  operation           text        NOT NULL,
  idempotency_key     text        NOT NULL,
  request_hash        bytea       NOT NULL,
  status              text        NOT NULL,
  resource_type       text,
  resource_id         text,
  response_code       integer,
  response_body       jsonb,
  created_at          timestamptz NOT NULL DEFAULT now(),
  expires_at          timestamptz NOT NULL,
  PRIMARY KEY (tenant_id, operation, idempotency_key)
);

The important behaviour is not the schema itself. It is the state machine around it.

A common pattern is:

1. compute canonical request hash
2. try to insert a started row for the key inside a transaction
3. if insert wins, this request owns execution
4. if the row already exists, compare hashes
5. if hashes differ, reject with conflict
6. if hashes match and status is completed, replay stored response
7. if hashes match and status is started, either wait, return 409, or return 202 with polling instructions depending on the API contract
8. when work finishes, update the row to completed with stored response payload

The first insert must be atomic. That is why databases, Redis scripts, or other compare and set capable stores are used. Two workers must not both decide they are the first.

Pseudocode shape:

const hash = canonicalHash(req.body)
const key = req.headers['idempotency-key']
 
const existing = await reserveOrLoad(key, hash)
 
if (existing.kind === 'hash_mismatch') {
  return conflict('key reused with different parameters')
}
 
if (existing.kind === 'completed') {
  return replay(existing.responseCode, existing.responseBody)
}
 
if (existing.kind === 'in_progress') {
  return tooEarly('original request still running')
}
 
const result = await executeBusinessOperation()
await markCompleted(key, result)
return result

The journal must be durable because the worst failures are crash boundaries. If the duplicate record lives only in process memory, a restart erases the only evidence that the operation already happened.

The journal should also store the original outcome that later retries need to see. If the first attempt returned 201 with paymentId: pay_712, a later retry should not fabricate a different paymentId or a different status code. The point is semantic replay, not just duplicate suppression.

Safe Retries Mean Replaying the Original Outcome, Not Re-running the Business Logic

The phrase deduplicate request can hide an important design choice. When a repeated request arrives, what exactly should the server do?

For completed operations, the best answer is usually: return the same outcome as the first successful completion.

That means:

• same semantic status
• same resource identifier
• same externally visible business result
• ideally the same response body shape, or one that is clearly documented as equivalent

If the first POST /payments produced:

{
  "paymentId": "pay_712",
  "status": "authorised",
  "authCode": "481922"
}

then the retry should return that same payment object, not create pay_713 and not answer with a vague "duplicate ignored" unless the API contract explicitly says so.

Why? Because clients are often written to continue from the returned object. If the first response was lost, the retry is how the caller learns what happened. A replayed result turns ambiguity into stability.

Handling in-flight duplicates

The harder case is a duplicate that arrives while the first attempt is still running.

There is no universal right answer. Common options are:

• wait and coalesce: the second caller blocks until the first finishes, then gets the same result
• return 409 or 425 style conflict: tells the client that work for this key is already in progress
• return 202 Accepted: caller can poll a status resource

The right choice depends on latency profile and client behaviour. For a short payment authorisation path, waiting briefly can be reasonable. For a ten minute provisioning workflow, a status handle is better.

Handling deterministic failures

Suppose the request is invalid because currency is missing. Repeating it with the same broken body should keep failing. There is no reason to run the business logic again.

Suppose the downstream rail returned a transient 503 before any side effect took place. Repeating later may succeed. Whether you cache that failure or allow retry depends on how confidently you know the effect did not happen.

This is why some implementations separate:

• validation failures
• pre effect transient failures
• post effect completed outcomes
• ambiguous internal failures that need operator review

What you must not do is treat every exception the same. The question is always: what does the system know about the underlying effect?

Key mismatch should be loud

If the same key arrives with different parameters, the server should not guess the caller's intent. It should reject clearly.

Example response:

HTTP/1.1 409 Conflict
Content-Type: application/json
 
{
  "error": "idempotency_key_reused_with_different_parameters"
}

Silent acceptance here is a correctness bug. It can bind one user's retry to another operation accidentally, especially in multi tab or mobile reconnect scenarios.

Idempotency therefore is not "do nothing on duplicates". It is: recognise one logical operation, preserve its first real outcome, and make later repeats observe that same truth.

The Guarantee Ends Early if Downstream Systems Cannot Also Deduplicate

A common failure mode is building idempotency correctly at the API edge and then losing the guarantee one hop later.

Example flow:

1. client sends POST /orders with key K
2. order API deduplicates correctly and creates one order row
3. order API publishes order.created twice because the worker crashed after publish acknowledgement but before marking the outbox row
4. warehouse consumer processes both messages and allocates stock twice

The API was idempotent. The overall system was not.

This is why downstream propagation matters.

Business keys should travel with events

If the payment pay_712 exists because of idempotency key K, that fact should not disappear immediately after the HTTP boundary. Events, jobs, and downstream requests should carry stable identifiers such as:

• payment ID
• order ID
• source request ID
• event ID
• idempotency key when appropriate

Consumers can then reason about whether they already processed the same logical instruction.

The outbox pattern protects event publication

If local state changes and event publication are separate steps, a crash between them causes loss or duplicate ambiguity.

The usual fix is the outbox pattern:

BEGIN;
 
INSERT INTO payments (...);
INSERT INTO outbox_events (event_id, topic, aggregate_id, payload) VALUES (...);
 
COMMIT;

A dispatcher later reads the outbox and publishes. If publication is retried, the event carries a stable event_id that consumers can deduplicate.

Consumers need an inbox or processed-message table

On the consumer side, message brokers are often at least once. The broker may redeliver if the consumer crashes after doing work but before acking.

A common consumer guard looks like this:

CREATE TABLE processed_messages (
  consumer_name text NOT NULL,
  message_id    text NOT NULL,
  processed_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer_name, message_id)
);

Consumer flow:

1. begin transaction
2. try to insert (consumer_name, message_id)
3. if insert fails on uniqueness, stop because this message was already handled
4. apply domain change
5. commit transaction
6. ack broker

Now a redelivery replays the same message ID and gets suppressed safely.

External vendors complicate the picture

Sometimes the downstream system is not yours. An email provider may happily send the same message twice unless you supply a dedupe reference. A shipping carrier may create a second label if you retry without a merchant reference. A bank rail may use its own end to end ID field.

Good integrations map their idempotency model onto the provider's own duplicate controls instead of pretending the first hop solved everything.

The practical rule is blunt: the logical operation needs stable identity all the way to the component that produces the irreversible side effect.

Queue Consumers Live in an At-Least-Once World, so Idempotency Is a Normal Part of Consumption

API retries get more attention because users can see them. Queue retries are just as important.

Most message systems choose availability over exactly once execution. A broker redelivers when acknowledgements are lost, workers crash, consumer groups rebalance, or visibility timeouts expire. That is healthy. It means the system prefers replay to silent loss.

But replay pushes duplicate responsibility onto the consumer.

Consider a stock reservation worker in Rotterdam handling order.confirmed events. It reads a message, decrements available stock, then the process dies before acknowledging. The broker redelivers. If the worker has no duplicate guard, one order reserves stock twice.

Consumer side idempotency usually relies on one or more of these:

• unique business identifiers such as reservation_id
• processed message tables keyed by message_id
• version checks on projections
• compare and set transitions in the target aggregate

A simple projection updater may accept only monotonic versions:

if event.version <= current.version:
    ignore as duplicate or stale
else:
    apply and set current.version = event.version

This is common when building read models from event streams. The projection is not saying the duplicate never existed. It is saying the state represented by version 14 has already been applied, so version 14 again should not change anything.

The same principle appears in ledgers. A posting engine may accept only one booking for (source_system, source_reference, posting_type). If the queue redelivers the same instruction, the uniqueness rule turns it into a no-op at the business layer.

One subtle point matters here: consumer idempotency and API idempotency are related but not identical.

• API idempotency is often about a client retrying because it never got a final answer.
• consumer idempotency is often about the platform replaying a message because delivery is at least once.

The data structures may look similar. The operational triggers differ.

What they share is the same correctness target: repeated delivery of the same logical input must not create repeated side effects.

Key Expiry, Scope, and Parameter Canonicalisation Decide What the Key Really Means

Once a team accepts that idempotency needs a key, the next mistakes are usually about meaning rather than storage.

The first question is scope. Is the key unique globally, per merchant, per endpoint, or per operation class? A raw key like 4819 is almost useless on its own. merchant_144:create_payment:4819 is far better because it says which caller and which action own the namespace.

This matters in multi-tenant systems. If two merchants both happen to send order-4819, they should not collide. If one merchant uses the same suffix for create-payment and create-refund, those should not collide either. The duplicate guard should merge only repeats that are truly the same logical operation.

The second question is how the server decides that two payloads are the same. A plain body string comparison is often too fragile. JSON field order, insignificant whitespace, and optional fields can vary between SDKs. A good implementation normalises the relevant fields into a canonical representation and hashes that canonical form.

For example, these two bodies should probably mean the same thing:

{"orderId":"ord_4819","amount":4800,"currency":"EUR"}

{"currency":"EUR","amount":4800,"orderId":"ord_4819"}

But these should not:

{"orderId":"ord_4819","amount":4800,"currency":"EUR"}

{"orderId":"ord_4819","amount":5800,"currency":"EUR"}

That sounds obvious, yet canonicalisation bugs cause real trouble. If one service hashes the raw JSON bytes and another hashes a normalised structure, the same SDK may appear idempotent on one path and mismatched on another. If optional defaults are filled in at different layers, the comparison can drift again.

The clean rule is to define exactly which fields participate in idempotency equivalence and canonicalise them once in the write boundary.

The third question is expiry. Teams often choose a retention window by guessing. That is risky because the window defines when the same logical request stops being recognisable.

Suppose a courier booking API retains keys for 6 hours. A warehouse client in Milan times out, queues a retry locally, and the site stays offline until the night shift restores connectivity 10 hours later. The retry reuses the same key, but the server has already evicted the record. If the original label was created, the late retry can now create a second label unless another business uniqueness rule catches it.

So the expiry window should come from observed client behaviour and business risk, not from a round number that felt tidy in a design meeting.

Questions worth answering explicitly:

• how long can legitimate retries arrive after the first attempt?
• do mobile devices or branch terminals buffer requests offline?
• can operators replay customer jobs the next day?
• is there another business uniqueness rule that survives beyond key retention?

In practice, strong systems often layer the protection:

• a short to medium retention window for fast idempotency replay
• a longer lived business uniqueness rule such as one payment per merchant_id + order_id + operation_type

The replay store and the business invariant solve different problems. The first gives good retry ergonomics. The second protects the domain when the retry arrives too late for the hot cache or journal.

Parallel Retries and Stuck In-Flight Records Need Their Own State Machine

Not every duplicate arrives after the first request has neatly finished. Many duplicates arrive while the original request is still running.

Examples:

• a user taps the pay button twice because the UI froze
• a mobile SDK retries aggressively while the first connection is still in flight
• a load balancer retries to another upstream after a short deadline
• two workers race to process the same queue message after a rebalance

At that point the idempotency record is not completed. It is something like started, pending, or in_progress.

That state needs a policy. Otherwise the system solves yesterday's duplicate and creates a new one under concurrency.

One policy is coalescing. The first request becomes the owner. Later requests with the same key wait for the owner to finish, then receive the same result. This gives a clean client experience when the work is short, but it ties up connections and requires careful timeout handling.

Another policy is explicit in-progress response. The service replies with a documented status such as 409 Conflict, 425 Too Early, or 202 Accepted plus a polling handle. This is often better for slower workflows such as account provisioning or report generation because it stops callers from sitting on long open sockets.

Whatever policy you choose, the system still needs to avoid stranded state. Suppose a worker inserts status = started, calls a dependency, and then crashes. If no recovery path exists, every later retry sees the in-progress row and is rejected forever.

This is why mature implementations often add one of these:

• a lease timestamp that must be renewed while work is active
• an owner identifier plus heartbeat
• a transition to unknown or needs_review when the lease expires
• a recovery path that queries downstream state before deciding whether to rerun

For a payment path, recovery may look like this:

1. retry sees stale started row
2. service checks whether a downstream authorisation reference was already recorded
3. if yes, it finalises the local row to completed and replays
4. if no, it safely re-executes or routes for operator review

That recovery logic is not optional polish. It is what turns the idempotency table from a lock graveyard into a correctness tool.

It also explains why many systems capture more than just completed or failed. Useful states can include:

• started
• completed
• failed_before_effect
• unknown_after_effect
• expired

The exact labels vary, but the distinction matters. failed_before_effect usually means a retry may run the business logic again. unknown_after_effect means the system must inspect other evidence before doing anything dangerous.

This is the same reason operators need good references into downstream systems. If a PSP returns auth_id=au_90812 and the server crashes before marking the idempotency row completed, that downstream reference may be the only clue that the first attempt already succeeded.

Parallelism therefore changes the question from "have we seen this key before?" to "what phase of this logical operation are we in, and what evidence do we have about side effects so far?"

That is a richer state machine than most beginner explanations admit, but it is the state machine real systems need.

Backoff, Jitter, Deadlines, and Retry Budgets Stop Retries from Becoming Self Harm

Once retries are safe, they still need to be polite.

A system that retries every transient failure immediately can turn a small outage into a load avalanche. This is especially dangerous during overload, partial network loss, or dependency brownouts.

Imagine 20,000 clients all making a write request at 09:00. The dependency behind the API starts returning 503 for 4 seconds. If every client retries after a fixed 200 ms pause, they align into fresh waves and keep the service pinned.

This is why good retry design includes:

• bounded retry conditions
• exponential backoff
• jitter
• overall deadline
• retry budget

Retry only the classes that make sense

Typical candidates:

• connection reset before a response
• timeout waiting for response
• 429 Too Many Requests
• 503 Service Unavailable
• occasionally 502 or 504 depending on gateway topology

Typical non candidates:

• 400 Bad Request
• 401 Unauthorized
• 403 Forbidden
• key mismatch conflicts
• domain declines such as insufficient funds where retrying the identical request immediately changes nothing

Use exponential backoff with jitter

A simple pattern:

attempt 1: wait random(0ms, 200ms)
attempt 2: wait random(0ms, 400ms)
attempt 3: wait random(0ms, 800ms)
attempt 4: wait random(0ms, 1600ms)

The randomisation matters. Without it, every well behaved client becomes synchronised.

Keep an overall deadline

Retries should live inside a total time budget. If the user action can tolerate 8 seconds, do not let a library keep retrying for 90 seconds in the background unless the product explicitly wants asynchronous completion.

Budget retries per request path or dependency

If a service is already failing heavily, unlimited retries amplify the failure. A retry budget caps how much extra traffic failures can create.

Client pseudocode:

for (let attempt = 1; attempt <= 4; attempt += 1) {
  const result = await sendOnce()
 
  if (result.ok) return result
  if (!isRetryable(result)) return result
  if (deadlineExceeded()) return result
 
  await sleep(randomBetween(0, 200 * 2 ** (attempt - 1)))
}

For idempotent writes, these transport rules and the server side duplicate guard reinforce each other.

• the server guarantees that repeated attempts converge on one business effect
• the client spreads those attempts out so the system can recover

Without the first property, retries are dangerous. Without the second, retries are abusive.

The Observability Has to Explain Ambiguous Outcomes After the Fact

When retries go wrong, operators do not ask abstract questions. They ask specific ones.

• Did the customer send one logical request or two?
• Did the first attempt reach the server?
• Did the side effect commit?
• Was the response lost?
• Did the retry reuse the same key?
• Did the key expire before the retry?
• Which downstream consumers saw the event?
• Was a duplicate suppressed, replayed, or actually applied twice?

If your logs and metrics cannot answer those, you do not really operate idempotency. You hope for it.

Useful log fields often include:

trace_id
request_id
idempotency_key
tenant_id
operation
request_hash
attempt_number
idempotency_status   // new, in_progress, replay, mismatch, expired
resource_id
response_code
downstream_reference

Useful metrics include:

• idempotency new requests per endpoint
• idempotency replay hits per endpoint
• hash mismatch conflicts
• in progress duplicate collisions
• expired key retries
• ambiguous timeout rate
• downstream duplicate suppression hits
• outbox replay count

A support runbook for a duplicate charge incident should be able to reconstruct a chain like this:

logical request key K created payment pay_712
client timed out after 3s
retry with same key K arrived after 800ms
server replayed original 201
webhook delivery for pay_712 retried twice
merchant consumer suppressed duplicate event evt_918

Or, if the system failed:

logical request key missing
first attempt created charge ch_991
response lost
client retried with fresh key
second attempt created charge ch_992
operator issued compensating refund rf_201

Those are very different incidents. Without good telemetry, they both collapse into "customer says they were charged twice".

Observability also lets you tune the design. If expired key retries are common, the retention window may be too short. If in progress collisions spike, clients may be firing parallel retries too aggressively. If replay hits are high on one mobile path, radio conditions or app deadlines may need work.

Retries and idempotency are therefore not only correctness mechanisms. They are measurement surfaces for hidden network and product behaviour.

The Smallest Useful Mental Model

A retry safe system does not promise magic. It does a few concrete things well.

1. It treats timeouts and lost responses as ambiguity, not proof of failure.
2. It gives each logical write operation a stable identity that survives transport retries.
3. It places a durable duplicate guard before or around the real side effect.
4. It binds that guard to request parameters so one key cannot silently mean two different things.
5. It stores enough completion state to replay the original outcome, not just block the second attempt.
6. It carries stable identifiers into queues, events, and downstream integrations.
7. It makes consumers idempotent because brokers and workers will replay.
8. It uses backoff, jitter, deadlines, and retry budgets so recovery traffic does not become the outage.
9. It emits logs and metrics that let operators prove what happened.

If you remember one sentence, make it this one: idempotency is how a distributed system turns repeated delivery attempts into one logical business effect.

That is why the topic keeps appearing in payments, ledgers, provisioning APIs, queue workers, webhook receivers, warehouse systems, and every other path where the cost of doing the same thing twice is higher than the cost of waiting a moment and being certain.