20-05-2026

How Webhooks Actually Work

Try the interactive lab for this article Take the quiz (6 questions · ~5 min)

Webhooks are often described as "HTTP callbacks". That description is correct and still not enough.

A real webhook system is an outbound event delivery pipeline sitting between one service's source of truth and another service's ingestion path. It has to turn internal state changes into durable events, sign them, send them over an unreliable network, survive timeouts, retry safely, handle duplicates and out-of-order delivery, and give both sides enough logs to debug failures days later. If any of those pieces are weak, the integration works in the demo and fails in production.

That is why webhook bugs rarely look like ordinary API bugs. The provider thinks it sent the event. The receiver thinks it never arrived. Both are right from their own vantage point. The provider retries and the receiver accidentally ships the same order twice. A signature check fails because the framework parsed and re-serialised the JSON before verification. A payment.succeeded event arrives before payment.processing because retries changed the order. An endpoint returns 200 OK before writing to disk, crashes a millisecond later, and quietly drops the event forever.

This article explains webhooks as distributed systems plumbing. We will look at how providers create events transactionally, what a signed delivery request actually contains, why the receiver must verify the raw body and persist first, how idempotency is implemented in practice, why ordering is often weaker than people think, how retries and dead-letter queues are designed, and what the security boundary really is when your application accepts commands from someone else's server.

A Webhook Is Provider-Initiated Event Delivery, Not A Reversed API Call

A normal API call is initiated by the client. Your application decides when to make an HTTP request, waits for the response, and owns the retry policy.

A webhook flips that. The provider decides when an event exists and when to deliver it. Your server becomes the receiver of an event stream that you do not schedule and cannot fully control.

That difference matters because the provider is not calling your endpoint merely to compute a response. It is informing you that something already happened in its own source-of-truth system.

Examples are everywhere:

a payment provider tells a merchant that an authorisation moved from processing to succeeded
GitHub tells a CI service that a push arrived on a repository
a payroll platform tells an ERP that a payout batch completed
an identity provider tells an application that a user account was deactivated

The provider therefore has two separate responsibilities.

commit the business event in its own system
attempt to deliver a representation of that event to subscribers

Those responsibilities must not be confused. If a payment provider marks a payment as captured in its internal ledger, the capture is real whether or not your webhook endpoint is online. The webhook is a delivery mechanism for that fact, not the fact itself.

This is why good webhook designs include a stable event identifier and usually an API to fetch the canonical current state. The receiver must be able to say, "I got event evt_0187, but I want to confirm the current resource state before acting." Webhooks are notification plus evidence, not necessarily the only source of truth.

It also explains a common anti-pattern: treating the browser redirect or front-end callback as authoritative. Suppose a checkout flow sends the customer back to https://merchant.example/return. That redirect is a user-experience signal. The customer's browser can close, stall, or be intercepted by extensions. The real state change lives in the provider's backend and is usually confirmed through the webhook channel or a follow-up API query.

If you remember one mental model from this article, make it this one: a webhook is not a fancy request-response endpoint. It is event delivery from one committed system state into another system's queue.

The Provider Must Create Events Transactionally, Or It Will Lose Them

The provider's first hard problem is not HTTP. It is event creation.

Imagine a billing platform in Dublin that stores invoices in PostgreSQL. A customer payment succeeds. The platform updates the invoice row from pending to paid and then intends to send a invoice.paid webhook.

If the code does this in the wrong order, bad things happen.

If it sends the webhook before the transaction commits, the receiver may observe a state that the provider later rolls back.
If it commits the transaction and only afterwards inserts a webhook job in a separate non-transactional step, a crash between those two operations loses the event permanently.

The standard fix is the outbox pattern.

Inside the same database transaction that changes business state, the provider also writes an outbox record describing the event to publish later.

BEGIN;
 
UPDATE invoices
SET status = 'paid', paid_at = now()
WHERE id = 481922;
 
INSERT INTO webhook_outbox (
  event_id,
  topic,
  aggregate_type,
  aggregate_id,
  payload,
  created_at
) VALUES (
  'evt_0187',
  'invoice.paid',
  'invoice',
  '481922',
  '{"invoiceId":481922,"status":"paid"}',
  now()
);
 
COMMIT;

Now the business fact and the event record succeed or fail together.

A background dispatcher can later poll the outbox table, publish each record into a delivery queue, mark attempts, and retry independently of the original request path. The customer-facing API no longer needs the webhook send to complete inline. That is important because outbound delivery is slow and unreliable compared with local database writes.

Some systems use database triggers to populate the outbox. Some do it in application code. Some read the write-ahead log or binlog directly and turn committed changes into events through CDC. The implementation choices vary, but the requirement is consistent: webhook events must be born from committed state, not from hope.

Once you adopt that model, other design choices become easier.

Event IDs can be created when the transaction commits.
Replay tools can read old outbox rows.
Failed deliveries do not threaten the primary write path.
Operators can inspect event history independently of delivery attempts.

Providers that skip this discipline often end up with the worst kind of integration incident: the internal system shows an order as shipped, but no corresponding webhook was ever recorded, so there is nothing to retry and nothing to prove.

A Delivery Attempt Is An HTTP Request With Identity, Time, And Proof Attached

Once an event exists, the provider has to package it into a delivery attempt. The body is only part of that story.

A serious webhook request usually needs at least these pieces:

a stable event identifier
an event type or topic
a creation timestamp
a delivery timestamp or signing timestamp
a signature or MAC over the raw payload and timestamp
a delivery identifier or attempt identifier for logging

A simplified request might look like this:

POST /webhooks/billing HTTP/1.1
Host: merchant.example
Content-Type: application/json
User-Agent: ExamplePay-Webhooks/1.0
Webhook-Id: evt_0187
Webhook-Topic: invoice.paid
Webhook-Timestamp: 1779175942
Webhook-Signature: v1=4c0a0d88d82e5db2f7d1e8bf2f9e1c5f98ad51d7f5f5a7ce7d8c6f0e4a112233
 
{"id":"evt_0187","type":"invoice.paid","data":{"invoiceId":481922,"status":"paid"}}

The key design idea is that the receiver must be able to answer three questions.

Which event is this?
When was this signed or sent?
Can I cryptographically verify that the provider created this exact byte sequence?

Most modern webhook systems use HMAC over a canonical signing input that includes the timestamp and raw body. For example:

signed_payload = timestamp + "." + raw_body
signature = HMAC_SHA256(secret, signed_payload)

The provider sends the timestamp and one or more signatures in headers. The receiver recomputes the HMAC with its copy of the shared secret and compares it in constant time.

Why include the timestamp? To limit replay. If an attacker steals a valid webhook request and can resend it forever, signature verification alone is not enough. A receiver normally accepts only signatures whose timestamp falls within a small tolerance window, such as five minutes, while still relying on event ID deduplication for legitimate provider retries.

The exact header names vary. Stripe, Svix, GitHub, Slack, and others all use slightly different conventions. The mechanism stays the same: identity, freshness, and proof travel with the request.

Another important detail is content canonicalisation. The signature covers bytes, not abstract JSON meaning. If the provider signs one string and the receiver verifies a re-serialised version with different whitespace or key ordering, verification fails. This is why webhook handlers so often need access to the raw request body before any JSON parser touches it.

Event Payload Design Is Part Of The Contract

The delivery mechanics are only half the design. The payload itself is a long-lived contract between provider and receiver, and sloppy payload design creates years of migration pain.

The first rule is stable identity. A webhook payload should carry:

a unique event ID
a stable resource ID or aggregate ID
an event type
the time the event was created
enough version or state information for the receiver to reason about ordering

That sounds obvious, but many weak webhook APIs still emit bodies that are hard to reconcile later because they only contain a topic name and a blob of loosely documented fields. If support receives a complaint from a retailer in Antwerp saying an order was shipped twice, the event log needs enough identity to trace one exact event through creation, delivery, deduplication, and business processing.

The second rule is deliberate choice between thin events and fat events.

Thin events carry identifiers and minimal metadata, expecting the receiver to fetch current state from the provider API.

Fat events carry a fuller snapshot of the resource so the receiver can act without another round trip.

Thin events reduce payload size, keep sensitive data concentrated in one API, and make the provider's schema easier to evolve. They also increase dependency on follow-up API calls and can make outages noisier because one webhook now needs two network paths to succeed.

Fat events make receivers simpler and reduce read amplification, but they increase payload churn and widen the blast radius of schema changes. They also raise privacy stakes because more customer data is now copied into more subscribers' logs and queues.

Most mature platforms end up in the middle. The payload is rich enough for common handling paths, but still carries a stable resource identifier so the receiver can fetch canonical state when the event is sensitive or ambiguous.

Versioning matters too. Event names often become part of customer code and cannot be renamed casually. A provider that starts with invoice.paid and later needs richer semantics has to decide whether to:

extend the payload compatibly
add optional fields and document defaults
publish a new topic version such as invoice.paid.v2
or expose per-endpoint API version pinning so old consumers keep the older shape

There is no universal right answer, but pretending versioning is unnecessary is the wrong one.

Another useful pattern is making the transition explicit. Instead of only saying "here is the resource after change," a payload may include the previous status and new status, or a monotonic sequence number, or an update counter. That helps the receiver decide whether it is seeing an old event, a duplicate, or a real new transition.

For example:

{
  "id": "evt_0187",
  "type": "invoice.paid",
  "created": "2026-05-20T14:03:11Z",
  "resource": {
    "id": "inv_481922",
    "version": 9,
    "previousStatus": "open",
    "status": "paid"
  }
}

That extra structure is not decoration. It gives the receiver a basis for idempotency and ordering decisions.

Finally, payloads should be tenant-safe and minimally scoped. If a provider serves many merchants, every event should carry the account or workspace context needed to prevent cross-tenant confusion. The receiver should never have to infer tenant identity from a free-form string or from the endpoint path alone.

Webhook bodies live for a long time in customer code, support tools, and archived logs. Treating them as a real API contract, not an internal struct dumped into JSON, pays for itself quickly.

Provider Delivery Fleets Need Isolation, Rate Control, And Replay Tooling

So far the webhook provider has sounded like one neat queue feeding one neat HTTP client. Real providers usually run a fleet of workers handling thousands or millions of destinations with wildly different behaviour.

One customer endpoint in Lisbon may answer in 40 milliseconds and never fail. Another in Bucharest may return 5xx for hours after every Friday deploy. A third may be rate-limited behind a shared ingress and accept only a handful of concurrent requests. If the provider pushes all of those destinations through one undifferentiated worker pool, the bad consumers become noisy neighbours for the good ones.

This is why provider-side delivery systems usually need isolation controls such as:

per-destination concurrency limits
per-tenant queues or partitions
circuit breakers for repeatedly failing endpoints
retry backoff with jitter
worker sharding so one hot customer does not monopolise all send capacity

Imagine a commerce platform with 40,000 subscribed endpoints. A sales event causes one large marketplace to emit two million order updates in fifteen minutes. If webhook jobs are just appended to one global FIFO queue, smaller tenants can experience huge delays behind one customer's burst. Better designs partition by tenant, destination, or topic family so throughput remains fair enough across the fleet.

Rate control matters for another reason: the provider must protect receivers from accidental abuse while still draining backlog. If an endpoint has been offline for six hours, the provider may have thousands of queued events for it. Dumping them all immediately when it comes back can cause another outage. Smarter fleets ramp carefully, perhaps with a small concurrency window that grows only after consecutive successes.

Circuit breakers help here. After a threshold of repeated failures, the provider can mark the destination degraded, slow the retry cadence, surface alerts in the dashboard, and preserve worker capacity for healthier endpoints. This is not just kindness. It is fleet hygiene.

Replay tooling is another first-class feature. Support engineers need to answer questions like:

resend only the failed deliveries for endpoint ep_1440 between 13:00 and 14:00 CEST
replay event evt_0187 with the current active secret
inspect the exact response bodies from the last five failed attempts
pause deliveries for one destination without pausing the entire tenant

Without these controls, every integration failure escalates into ad hoc database queries and manual scripts, which is exactly the kind of operational improvisation that causes secondary incidents.

Good fleets also separate event generation time from delivery time in their metrics and customer UI. If a receiver sees an event at 14:20 but it was created at 14:03, the difference is diagnostic gold. It tells you whether the problem sat in the provider queue, the network, or the receiver's own processing path.

At scale, webhook delivery stops being a simple HTTP problem and becomes a scheduling problem. Which jobs should run now, how hard should each destination be pushed, how much backlog is acceptable, and how can operators replay safely without duplicating side effects? The provider that answers those questions well is the provider whose webhooks feel boring for everyone downstream.

The Receiver Must Verify The Raw Body Before It Trusts Anything

On the receiver side, the first danger is accepting untrusted HTTP as if it were an internal command. A webhook endpoint sits on the public internet. Anyone can send it packets. The point of verification is to distinguish the provider's real events from forged traffic.

The safe processing order is usually:

read raw request bytes
extract timestamp and signature headers
reconstruct the provider's signing input exactly
compute the HMAC or signature verification locally
reject if the signature fails or the timestamp is outside tolerance
only then parse the body as JSON and continue

In Node or TypeScript pseudocode, the shape is usually something like:

const rawBody = await getRawRequestBody(req)
const timestamp = req.headers['webhook-timestamp']
const signature = req.headers['webhook-signature']
 
if (!timestamp || !signature) {
  return new Response('missing headers', { status: 400 })
}
 
const signedPayload = `${timestamp}.${rawBody}`
const expected = hmacSha256Hex(secret, signedPayload)
 
if (!constantTimeEqual(expected, extractV1(signature))) {
  return new Response('invalid signature', { status: 400 })
}
 
const ageSeconds = Math.abs(currentUnixSeconds() - Number(timestamp))
if (ageSeconds > 300) {
  return new Response('stale webhook', { status: 400 })
}
 
const event = JSON.parse(rawBody)

The constant-time comparison is not theatre. Straight string comparison can leak timing differences that help an attacker distinguish partial signature matches. The risk depends on the environment, but the safer primitive is easy to use, so it is the right default.

Signature verification also needs secret rotation discipline. Providers should support overlapping secrets so the receiver can accept signatures from both old and new secrets during rotation. Receivers should key secrets by endpoint or subscription, not by a single global string pasted into every integration.

A second trap is doing expensive or irreversible work before verification. If the handler parses the body, inserts rows, calls internal services, or sends emails before proving the request is authentic, the system gives unauthenticated internet traffic too much influence.

IP allowlists help less than people think. Large providers may send from changing address ranges, via CDNs, or from multiple regions. Allowlisting can reduce obvious noise, but it is not a replacement for message authentication because network origin is not the same thing as request authenticity.

Returning 200 Too Early Is The Fastest Way To Drop Events

After verification, the receiver has another crucial ordering problem. It should acknowledge success to the provider only after it has durably recorded the event somewhere that can survive a crash.

This is the mirror image of the provider's outbox problem.

If the receiver does this:

verify signature
enqueue work in memory or start asynchronous processing
return 200 OK
crash before the event is stored durably

then the provider believes delivery succeeded and may stop retrying, but the receiver has lost the event. That is silent data loss.

The safer pattern is an inbox table or durable queue.

INSERT INTO webhook_inbox (
  event_id,
  topic,
  received_at,
  payload,
  status
) VALUES (
  'evt_0187',
  'invoice.paid',
  now(),
  '{...}',
  'received'
)
ON CONFLICT (event_id) DO NOTHING;

Only after that insert commits should the handler return a 2xx response.

The provider's retry system then has a clean contract. A 2xx means the receiver has durably accepted the event. A timeout, 429, or 5xx means try again later. The provider does not need to know whether business processing finished. It only needs to know whether responsibility for the event has safely transferred.

This leads to a robust two-stage receiver architecture.

Ingress path: authenticate, deduplicate at the envelope level, persist durably, return 2xx.
Worker path: process persisted events asynchronously, call internal services, update business state, trigger side effects, retry locally if needed.

That split solves a lot of problems at once.

Slow downstream dependencies do not force the provider to wait on the HTTP request.
A temporary bug in one consumer does not force the entire webhook endpoint to stop acknowledging all traffic.
Operators can re-run workers against stored inbox events without asking the provider to resend everything.
Throughput can be scaled by adding workers without changing the external contract.

People sometimes resist this because it feels like extra indirection. It is extra indirection. It is also the difference between "webhooks usually work" and "webhooks remain recoverable under process crashes, deploy restarts, and database hiccups".

Idempotency Is The Receiver's Core Safety Property

Webhook delivery on the public internet is almost always at-least-once, not exactly-once.

The provider may send the same event again because:

the endpoint timed out
the provider did not receive the response cleanly
the receiver returned a retryable status
the provider's own worker crashed after send and before recording success
an operator manually replayed the event

Therefore the receiver must be able to process duplicates without duplicate side effects.

There are two related but different idempotency keys in webhook systems.

Event idempotency. The stable event ID, such as evt_0187, tells you whether this exact event envelope was already seen.

Business idempotency. Sometimes different event IDs still represent the same underlying business transition for your system. For example, you may only want one local shipment record for provider order ord_5541 even if a provider emits multiple related notifications.

A durable inbox with a unique index on event_id handles envelope-level deduplication. The first insert succeeds. Later duplicates do nothing.

CREATE UNIQUE INDEX webhook_inbox_event_id_key
ON webhook_inbox (event_id);

But that is not always enough. Suppose your worker reads the event, starts creating a local invoice, crashes halfway through, and the job is retried. You now need idempotency at the business operation level too. Maybe the local invoice table has a unique constraint on provider_invoice_id. Maybe the state transition function checks whether the target status is already present before applying it. Maybe both.

Exactly-once usually means one of two things in marketing copy.

duplicates are possible on the wire, but deduplication makes side effects idempotent
a scoped system, such as one queue partition with transactional consumption, gives stronger guarantees inside a narrow boundary

Across arbitrary providers, internet delivery, and your own application logic, assume at-least-once and design for idempotent outcomes.

A good worker therefore tends to look like this:

load persisted inbox record
check whether the business transition was already applied
apply it transactionally if not
mark the inbox record processed
make any external side effects themselves idempotent where possible

Without this discipline, duplicate delivery becomes duplicate shipping, duplicate account creation, or duplicate email storms. Webhooks are unforgiving on this point because duplicates are normal, not pathological.

Ordering Is Often Weaker Than You Want, So Fetching Current State Matters

Developers often assume that if event A happened before event B, webhook A will arrive before webhook B. That can be true on a quiet day and false the first time retries, sharding, or worker restarts appear.

Ordering breaks for several ordinary reasons.

The provider may dispatch different event types from different workers.
A slow first request can time out and be retried later, while a newer event succeeds immediately.
One delivery attempt may traverse a longer network path.
The receiver may persist events in order but process them with concurrent workers.

Suppose a subscription platform emits:

invoice.created
invoice.finalised
invoice.paid

If the first event's attempt gets a timeout, the receiver may see invoice.finalised and invoice.paid first. That does not mean the provider is broken. It means internet delivery is not a serial queue with perfect timing guarantees.

There are several ways to cope.

Monotonic versioning. The event payload includes a resource version or updated-at timestamp. The receiver applies only newer versions.

State fetch on important transitions. Instead of trusting every event as complete truth, the receiver uses the event as a trigger to fetch the provider's current resource state.

Per-aggregate serialisation. Some receivers route all events for one customer, invoice, or order key to the same worker partition so local handling preserves order better.

Transition guards. The local state machine refuses impossible backwards transitions unless explicitly allowed.

The fetch-current-state pattern is especially useful for external integrations. If you receive payment.succeeded, the safest action may be:

verify and persist the event
call the provider API for payment pay_4819
inspect the current canonical status
update your local record to match that state idempotently

This avoids building too much business logic on the arrival order of notifications.

Of course it increases API dependence and latency, so it is not free. Many systems compromise by fetching only on sensitive or ambiguous transitions while trusting less critical events directly.

What matters is not blind faith in order. It is having a model for what happens when order is imperfect.

Retries, Backoff, And Dead-Letter Queues Are Part Of The Protocol, Not An Afterthought

Since webhook delivery is unreliable, the provider needs a retry policy. A single send attempt is not a webhook system. It is a best-effort POST.

A typical retry design answers these questions.

Which status codes are terminal success?
Which failures should be retried?
How long should the provider wait between attempts?
When does the event stop retrying and move to a dead-letter state?
How can operators replay failed deliveries?

The usual rule is simple. Treat any 2xx as success. Treat timeouts and most 5xx responses as retryable. Treat some 4xx responses as terminal configuration errors, though providers differ here because certain receivers use 429 or transient 401 patterns during maintenance and secret rotation.

Backoff is normally exponential with jitter so a provider does not hammer a failing endpoint in a tight loop.

attempt 1: immediate
attempt 2: +30 seconds
attempt 3: +2 minutes
attempt 4: +10 minutes
attempt 5: +1 hour
attempt 6: +6 hours

The exact schedule varies, but the intent is stable. Give transient failures time to clear while keeping pressure low enough that one broken consumer does not exhaust the provider's delivery fleet.

Dead-letter queues or failed-delivery ledgers matter because not every receiver will recover by itself. A merchant may delete an endpoint, deploy a bug, or rotate a secret incorrectly. The provider needs durable evidence that event evt_0187 failed 17 times over 48 hours, with response codes and bodies recorded.

That evidence serves three roles.

customer support can diagnose integration failures
operators can replay events after the endpoint is fixed
auditing can prove whether an event was generated, attempted, and ultimately acknowledged

Receivers have a parallel obligation on their own side. Once an event is in the durable inbox, local worker retries should use their own backoff and poison-message handling. A malformed but authenticated event should not block the entire queue forever.

Another practical receiver pattern is separating permanent failure from temporary failure explicitly. If deserialisation fails because the payload is malformed relative to the documented schema, the event should usually be marked as bad input and surfaced to operators immediately. If processing fails because an internal dependency in Prague is timing out, the inbox worker should retry locally without asking the provider to generate a second external delivery. That distinction keeps the provider retry loop focused on network acceptance while the receiver's own queue handles downstream instability on its side of the boundary.

This is why mature webhook systems expose delivery logs. The right debugging view is not just the event payload. It is a timeline:

evt_0187 created 14:03:11 CEST
attempt 1 -> 500 in 142 ms
attempt 2 -> timeout at 10 s
attempt 3 -> 200 in 88 ms

Without that trail, every incident becomes a blame game between provider and receiver.

The Security Boundary Is HMAC, Freshness, Least Privilege, And Careful Parsing

Webhook security is sometimes reduced to "use HTTPS". HTTPS is required and nowhere near sufficient.

The full threat model includes at least these cases.

random internet traffic hits the endpoint
a valid request is captured and replayed later
a secret leaks from logs or configuration sprawl
a framework bug causes signature verification against altered bytes
an internal service trusts the webhook payload too broadly once it passes ingress

The usual HMAC pattern addresses authenticity and integrity. The timestamp tolerance addresses freshness. Deduplication addresses legitimate retries and many replay scenarios. But other controls matter too.

Least privilege per endpoint. Do not use one shared secret for every customer and every topic if you can avoid it. Scope secrets to subscriptions or destinations so one leak does not compromise the whole fleet.

Secret rotation. Providers should support a primary and secondary secret during rotation. Receivers should store activation windows and stop accepting the old one after a planned cutover.

No secret exposure in logs. Raw headers can easily leak signatures or secrets into log pipelines. Redact aggressively.

Payload validation after authentication. A valid signature means the provider sent the message, not that the payload is semantically safe. Validate schema, topic, account ownership, and expected field types before triggering side effects.

Tight internal authorisation. A verified webhook should not directly call high-privilege business functions without policy checks. For example, a user.deleted webhook for tenant A should not let the handler delete a user in tenant B because of a mapping bug.

Careful clock handling. Timestamp tolerance assumes reasonably sane clocks on both sides. If your receiver clock drifts badly, you may reject real requests. Monitoring time sync on webhook ingress hosts is therefore a security dependency.

There is also a subtle provider-side security issue: some providers include too much data in webhook payloads. If the receiver only needs a customer ID and status, sending full PAN-adjacent billing artefacts, personal addresses, or internal fraud annotations increases breach impact for no gain. Webhook payloads should be minimal representations of the event needed by the subscriber.

Security in webhooks is therefore message authentication plus data minimisation plus disciplined internal handling.

Good Webhook Systems Make Failure Visible And Recovery Routine

The final difference between toy webhook implementations and serious ones is operational visibility.

A serious provider keeps structured records for:

event creation time
subscription or endpoint chosen
each delivery attempt ID
request headers and body hash
response code and latency
retry schedule
final disposition

A serious receiver keeps structured records for:

when the event was first received
whether signature verification passed
whether it was a duplicate by event ID
when it entered the durable inbox
worker processing attempts and results
which local business object was affected

These records turn debugging from folklore into evidence.

They also make reprocessing possible. If a warehouse connector was down for two hours, you want to replay exactly the affected events, not ask the provider to generate a full account export because your webhook history is opaque.

Metrics help too. Useful ones include:

provider send rate by topic
provider success rate by destination
provider queue depth and oldest pending age
receiver verification failure rate
receiver duplicate rate
receiver inbox depth and processing latency
end-to-end delay from provider event creation to receiver business application

One metric deserves special attention: age of oldest unprocessed inbox event. Throughput averages can look fine while one poisoned partition leaves a specific customer's events stalled for hours.

Another is signature failure ratio after deploys. A sudden jump often means the receiver changed middleware ordering and lost access to the raw request body, or the provider rotated secrets incorrectly.

When all of this is in place, failure becomes survivable.

The provider can keep retrying politely. The receiver can acknowledge only after durable acceptance. Duplicates do not create duplicate business outcomes. Operators can inspect the trail. Specific events can be replayed. Out-of-order arrival becomes a designed-for condition instead of a mystery.

That is the difference between a webhook feature and a webhook system.

Webhooks Are Reliable Only When Both Sides Respect The Queue

At first glance, webhooks look simple because the transport is just HTTP. The complexity arrives from everything HTTP does not guarantee.

HTTP does not guarantee that the sender knows whether the receiver committed the event before the connection broke. It does not guarantee ordering across retries. It does not guarantee exactly-once delivery. It does not make a 200 OK meaningful by itself. Both sides have to build those meanings with logging, signatures, durable queues, idempotency, and replayable history.

The provider's job is to create events transactionally, sign and deliver them with evidence, retry responsibly, and expose enough telemetry for support and recovery. The receiver's job is to verify raw bytes, persist before acknowledging, process idempotently, and avoid treating arrival order as stronger than it is.

Once you see webhooks as queue-backed event transfer rather than just outbound POST requests, the correct designs stop looking optional. They become obvious consequences of the failure model.

That is why the best webhook implementations feel boring in production. They are not relying on luck. They are respecting the queue on both sides of the wire.