System Design: [Webhook delivery platform name]

Reference design for an outbound webhook platform that delivers event notifications from your product to endpoints that customers register: at-least-once delivery, payload signing, per-endpoint isolation and queues, retries with backoff and circuit breaking, ordering options, replay, endpoint verification, protection against server-side request forgery (SSRF) and delivery observability for customers. The signing and header format follows the Standard Webhooks specification v1.0.0; error bodies follow RFC 9457. It follows the structure of the System Design Document template and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative assumptions, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Webhooks are sent from the API servers in the request thread pool through one shared queue. When a large customer's endpoint started timing out after 30 s last month, worker threads filled up and deliveries to every other customer were delayed by up to 3 hours. Retries are a fixed 3 attempts over 10 minutes, so short customer outages lose events. Payloads are unsigned, and a penetration test showed that a registered URL pointing at the cloud metadata address returned internal credentials.]

1.2 Proposed solution

[Example: Product services publish domain events to an event log. A fan-out stage matches each event to subscribed endpoints and creates one delivery per endpoint. Deliveries are scheduled per endpoint, with a concurrency cap and a circuit breaker for each endpoint, so one slow endpoint only delays its own deliveries. Workers send signed HTTPS POST requests in the Standard Webhooks format through an egress proxy that blocks private and internal addresses. Failed deliveries are retried with exponential backoff and jitter for about 3 days, then marked failed and available for manual or API replay for 30 days. Customers see every attempt in a delivery log and are notified when an endpoint keeps failing.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

• [Example: inbound webhooks from third parties into our product]
• [Example: exactly-once delivery (impossible over HTTP without receiver cooperation; receivers deduplicate by event ID)]
• [Example: push notifications to devices, email and SMS (Notification Service System Design Template, system-design/design-notification-service)]
• [Example: event streaming to customer cloud accounts (separate connectors)]
• [Example: transforming payloads per customer]

3. Capacity estimation

3.1 Assumptions

3.2 Delivery rate

2 billion / 86,400 s = about 23,150 events/s on average, about 69,400/s at peak (round to 70,000). With 1.5 endpoints per event: about 105,000 deliveries/s at peak.

Consequence: a single queue partition or a single scheduler cannot keep up; partition the scheduler by endpoint ID (Example: 256 partitions, about 410 deliveries/s each at peak).

3.3 Concurrent requests

By Little's law, healthy traffic needs 105,000/s x 0.3 s = 31,500 requests in flight. If 1% of deliveries hit slow endpoints that run to the 10 s timeout, that adds 1,050/s x 10 s = 10,500. Total about 42,000.

Consequence: slow endpoints add a third of the in-flight load from 1% of traffic. Use non-blocking HTTP clients (Example: 2,000 in-flight requests per worker, so 21 workers at peak, plan 32 for headroom and zone loss) and cap in-flight requests per endpoint so a slow endpoint cannot take more than its share.

3.4 Failure backlog

Deliveries per day = 2 billion x 1.5 = 3 billion. If 2% go to failing endpoints, 60 million deliveries per day enter retry; held for up to 3 days that is up to 180 million pending deliveries.

Consequence: the pending set does not fit in a broker's memory and must not block healthy traffic. Store pending deliveries durably with an index on (endpoint_id, next_attempt_at) and let a circuit breaker stop attempts to dead endpoints (7.2) instead of retrying each delivery independently.

3.5 Storage

Payloads are stored once per event: 2 billion x 4 KB = 8 TB per day, 240 TB for 30 days before compression. Attempt records at Example: 200 bytes and 1.2 attempts per delivery: 3 billion x 1.2 x 200 B = 720 GB per day, about 21.6 TB for 30 days.

Consequence: keep payloads in compressed object storage keyed by event ID, and only metadata and attempt records in the database; replay reads the payload from object storage.

3.6 Summary

4. API design

4.1 Management API

Management API errors use Problem Details (RFC 9457); for example a rejected URL returns 422 with the reason (private address, not HTTPS).

4.2 Webhook request format

The request follows Standard Webhooks v1.0.0: three headers and a JSON body with type, timestamp and data.

POST /hooks/acme HTTP/1.1
Host: hooks.customer.example
Content-Type: application/json
webhook-id: evt_2Lq8x7Kp
webhook-timestamp: 1792656000
webhook-signature: v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDN/H/pI4=

{"type": "invoice.paid", "timestamp": "2026-10-22T08:00:00Z", "data": {"id": "inv_93Fa", "amount": 12000, "currency": "EUR", "status": "paid"}}

• webhook-id is the event ID and stays the same on every retry, so the receiver can use it as an idempotency key.
• webhook-timestamp is set per attempt, in Unix seconds.
• webhook-signature is v1, plus the base64 HMAC-SHA256 of webhook-id.webhook-timestamp.body using the endpoint secret. During rotation it carries two signatures separated by a space.
• The signature value above is illustrative, not a real computation.

Keep payloads small (the specification suggests staying well under about 20 KB); for large objects send an ID and let the receiver fetch the resource through the API.

4.3 Response handling

5. Data model

5.1 Entities

One event fans out to many deliveries; each delivery has one or more attempts.

5.2 Storage choices

5.3 Access patterns

6. High-level design

7. Detailed design

7.1 Deep dive 1: per-endpoint isolation

Choice: C. Reason: it gives each endpoint its own backlog and limits without creating broker objects per customer. Each scheduler partition:

• Keeps a ready set of endpoints with due deliveries, ordered by earliest due time.
• Takes deliveries in rounds across endpoints (deficit round robin), so a customer with a million pending events gets the same turn as one with ten.
• Enforces a per-endpoint in-flight cap (Example: 10 by default, raised for large customers on request) and a per-account cap.
• Never assigns more work to an endpoint whose breaker is open.

The delivery sequence for one healthy endpoint:

7.2 Deep dive 2: retries, backoff and circuit breaking

Retry schedule (Example, each delay with plus or minus 20% jitter):

Choice: B, then C after sustained failure. Reason: an endpoint is usually down as a whole, so its deliveries should wait together. The breaker opens after Example: 10 consecutive failures or more than 50% failures over 5 minutes, sends one probe delivery per backoff interval, and on success closes and drains the backlog at the endpoint's cap so recovery does not flood the customer. Deliveries still respect their own expiry: after about 3 days without success they become failed and stay replayable for 30 days. After Example: 5 days of continuous failure the endpoint is disabled and the customer is notified by email and in the dashboard, as the specification recommends.

7.3 Deep dive 3: ordering

Choice: A by default, C for endpoints that opt in. Reason: strict ordering turns any single failure into a full stop, and most receivers only need the latest state. Every payload carries the event creation time and a per-resource sequence number so receivers can discard older updates. In ordered mode the scheduler releases the next delivery for a key only after the previous one succeeded or expired, and the dashboard shows blocked keys.

7.4 Deep dive 4: signing, endpoint verification and SSRF

Signing (Standard Webhooks v1.0.0):

• One secret per endpoint, 24 to 64 random bytes, shown once with the whsec_ prefix; stored in a key management service.
• Symmetric signatures use HMAC-SHA256 (RFC 2104) with the v1 prefix; asymmetric signatures use ed25519 with the v1a prefix for customers who must not hold a shared secret.
• Sign the exact bytes sent; never re-serialise the body after signing.
• Rotation sends two signatures during the overlap period.
• Customer documentation tells receivers to compare signatures in constant time, reject timestamps outside a tolerance (Example: 5 minutes) to stop replays, and deduplicate by webhook-id.

Endpoint verification: on registration and on URL change, the platform sends a webhook.verification event containing a random challenge; the endpoint is activated only after a 2xx response that echoes the challenge. This proves the customer controls the URL and stops the platform from being used to send traffic to third parties.

SSRF protection (OWASP Server Side Request Forgery Prevention Cheat Sheet):

How real systems do it (paraphrased from Stripe's public webhook documentation; see 12.4). Stripe tells integrators that live-mode events are retried with exponential backoff for up to three days, that delivery order is not guaranteed and the same event can arrive more than once, so receivers should record processed event IDs. Each request carries a signature header with a timestamp and an HMAC-SHA256 signature over the timestamp and raw body; the official libraries reject timestamps older than a default tolerance of five minutes, and a new signature and timestamp are generated for each retry. Redirect responses count as failures. Secrets can be rolled with a period during which both old and new secrets produce signatures. Receivers are told to return a 2xx quickly and do heavy work asynchronously. Lesson for this design: publish the same plain contract (at least once, unordered, signed, retried for a stated period) and give receivers the tools to follow it.

7.5 Customer-facing observability and replay

• Delivery log per endpoint: event ID, type, attempt number, status code, latency, error class (timeout, TLS, DNS, refused, HTTP status) and a response excerpt.
• Endpoint health page: success rate over 24 hours, p95 latency, breaker state, backlog size and the time of the next attempt.
• Replay: by event ID, by filter, or by time range; replays create new deliveries flagged replay=true, keep the original webhook-id, and are rate-limited per endpoint.
• Notifications: email and dashboard banner when the breaker opens, again before the endpoint is disabled, and a summary when it recovers.
• Events API: customers can list recent events through the API to reconcile anything they missed, which is the safety net behind at-least-once delivery.

8. Scalability and performance

Latency budget for the first attempt to a healthy endpoint (target p99 5 s from event commit):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security

Run a threat model (Threat Model Template, security/threat-model) on the egress path and the fan-out account check before launch.

9.3 Observability (operator)

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Move accounts in cohorts from the legacy sender to the new pipeline. Signatures are added before the cutover with at least 60 days' notice so customers can enable verification. Rollback routes a cohort back to the legacy sender; deliveries already created in the new pipeline complete there.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Traceability

12.3 Glossary

12.4 References

Standard and licence note

Based on the Standard Webhooks specification v1.0.0 for the request format and signatures, on the OWASP SSRF Prevention Cheat Sheet for egress controls, and on RFC 2104, RFC 9110 and RFC 9457. Stripe material is used as inspiration only and is paraphrased, not copied. All numbers are illustrative examples. Check the current version of each specification before you implement. Original text written for this collection. Template licence: MIT.