Short answer: outbound webhooks are a public API with worse failure modes, because delivery depends on someone else's server. Treat them as a product surface: a versioned event contract, signed payloads, at-least-once delivery with idempotency keys, a documented retry schedule, a dead-letter path, a replay tool, and per-endpoint observability. If you cannot answer "what happened to event 8f3c, and can we send it again", your integration surface is not finished.
Outbound webhooks are a delivery promise: signed, retried, replayable and observable.
Why outbound webhooks deserve product treatment
Inbound webhooks are your problem, and you can fix them by improving your own code. Outbound webhooks are your customer's problem, and their endpoint will be slow, misconfigured, behind a corporate proxy, or down for maintenance at the worst possible time. Your product's reputation depends on how gracefully you handle that.
The symptoms of an unfinished webhook surface are recognizable: support tickets asking whether an event was sent, engineers querying the database by hand to find a payload, customers writing polling loops because they cannot trust delivery, and a silent backlog nobody notices until an invoice does not reconcile.
Design the event contract first
The contract is the part customers build against, so it needs the same discipline as any public API.
- Stable, past-tense, resource-oriented event names.
invoice.paid,subscription.cancelled,user.role_changed. Names are forever; get them right before the first customer. - An envelope with explicit fields. Event id, event type, API version, created timestamp, and the resource. Keep the envelope stable across event types so customer code can share a parser.
- Version in the envelope, not only in the URL. When the payload shape changes, customers need to know which shape they received without guessing from a timestamp.
- Reference data, do not duplicate it. Send identifiers and a minimal snapshot; let the customer fetch the full object through your API with their own credentials. Duplicating the whole object in every event makes versioning painful and creates stale copies.
- Document the ordering guarantees and their limits. "Events for the same resource are sent in order of occurrence; ordering across resources is not guaranteed" is honest and useful. Silence invites wrong assumptions.
Delivery states and semantics
Name the states explicitly, expose them to customers, and make each one observable.
| State | Meaning | What the customer should see |
|---|---|---|
| Pending | Accepted into the delivery queue | Nothing; internal only |
| Delivering | An attempt is in flight | Delivery log entry with attempt number |
| Delivered | Endpoint returned a success response | Delivered timestamp and response code |
| Retrying | Attempt failed with a retryable condition | Next attempt time, attempt count |
| Dead-lettered | Retries exhausted, or a permanent failure | Failure reason and a replay action |
| Replayed | Re-sent manually or by policy | New attempt entries linked to the original event |
Two distinctions prevent most confusion. First, permanent vs retryable failure: a 401 or 404 will not fix itself on the fifth attempt, so retrying is noise; a 500 or timeout usually will. Second, the difference between "we accepted it" and "your endpoint processed it" — customers need the second, and only the delivery log can show it.
Delivery guarantees and idempotency
At-least-once delivery is the honest default. Exactly-once is not achievable across a network boundary, so design for duplicates and make them harmless.
- Send an idempotency key with every event: a stable event id that does not change across retries or replays.
- Document the expectation explicitly: customers should treat an event id they have already processed as a no-op.
- Keep event ids unique per occurrence, not per resource, and never reuse an id for a different payload.
- For ordering-sensitive flows, include a sequence number or resource version so the customer can discard out-of-order deliveries.
Retries, backoff, and dead letters
A retry schedule should be documented, bounded, and jittered. A common shape is a handful of attempts over a few hours, with exponential backoff:
| Attempt | Delay after failure | Notes |
|---|---|---|
| 1 | Immediate | Initial delivery |
| 2 | ~1 minute | Jittered |
| 3 | ~5 minutes | Jittered |
| 4 | ~30 minutes | Jittered |
| 5 | ~2 hours | Last automatic attempt |
| — | Dead letter | Retained for a documented window, replayable |
Then answer the questions the schedule raises. How long is a dead-lettered event retained? Can the customer trigger a replay, and is it rate-limited? Does replay preserve the original event id (it should)? Does a successful replay clear the failure? Are replays audited?
A replay tool is not a nice-to-have. Without it, every delivery failure becomes a support ticket and a manual database operation, which is exactly the work that does not scale.
Signatures and secrets
- Sign every payload. An HMAC over a canonical representation, with a timestamp included to prevent replay.
- Publish the verification snippet in the customer's language of choice. Most integration failures are verification bugs, and a snippet removes a class of support tickets.
- Support secret rotation with two active secrets during a transition window, and document the overlap.
- Never put secrets in the payload. The signature proves authenticity; the payload should carry no credentials.
- Document what the signature covers. Headers, body, and timestamp — exactly, including whitespace handling, because the customer's verification must reproduce it byte for byte.
Observability and support tooling
The internal surface matters as much as the public one. Build:
- A per-endpoint dashboard: delivery success rate, p50/p95 latency, consecutive failures, and last success time.
- A per-event delivery log, visible to the customer, with attempt history and response codes.
- Automatic disablement or alerting when an endpoint fails consistently — after notifying the customer, and with a documented reactivation path.
- Alerts on queue depth and on events that exhaust retries, because a growing dead-letter queue is a silent outage.
- A correlation id that ties the originating product action to the event, so support can trace one complaint end to end.
Launch checklist
- Event names, envelope, and versioning documented in a public reference.
- Signature scheme documented with a verification snippet and rotation process.
- Retry schedule, dead-letter retention, and replay policy published.
- Idempotency and ordering guarantees stated in plain language.
- Delivery log visible to customers, with a replay action.
- Alerts on failure rate, queue depth, and dead letters, routed to an owner.
- A test endpoint or sandbox so customers can integrate before going live.
- A migration note for any future payload change, agreed before the first customer integrates.
Limitations and assumptions
This design assumes you control the delivery pipeline and can store delivery state; products that send events from a serverless function with no queue will need a small delivery service first. It assumes customers can expose an HTTPS endpoint — for those that cannot, offer polling as a documented fallback rather than leaving them to invent it. It assumes a manageable number of endpoints per tenant; at high fan-out, per-endpoint isolation and rate limiting become their own engineering problem. Finally, at-least-once delivery shifts responsibility to the customer for idempotency: if your documentation does not make that expectation unmissable, you will spend the savings on support.
Working with CodeAustral
We design and build integration surfaces — versioned events, signed delivery, replay tooling, and observability — as part of the platform work behind B2B products. If your customers are asking for webhooks or building polling loops around your API, see how we build product platforms or send a brief, and read our notes on processing inbound webhooks reliably for the other side of the same problem.
Frequently asked questions
Should we use webhooks or polling for our customers?
Offer webhooks as the primary mechanism and polling as a documented fallback. Polling is simpler for a customer to build but scales badly for both sides, and it turns every delay into a support question. If you offer both, document which events are available through each and keep the semantics consistent.
How many retry attempts are enough?
Enough to survive a typical maintenance window on the customer's side, and few enough to keep the dead-letter queue meaningful. Five to eight attempts spread over a few hours is a common shape. The important part is documenting the schedule and making failures visible, not the exact numbers.
Do we need to support replay?
Yes. Endpoints break, customers fix them, and without replay the only recovery is a manual database operation. A rate-limited, audited replay action turns a recurring support cost into a self-service feature.
How do we handle a customer whose endpoint is always failing?
Notify them, keep the events in the dead-letter store for the documented window, and consider automatically pausing delivery after a sustained failure to protect your own queue. Pausing without notification is worse than failing loudly, so the alert and the notification belong in the same flow.