# System Design: [Webhook delivery platform name]

> Reference design for an outbound webhook platform that delivers event
> notifications from your product to endpoints that customers register:
> at-least-once delivery, payload signing, per-endpoint isolation and
> queues, retries with backoff and circuit breaking, ordering options,
> replay, endpoint verification, protection against server-side request
> forgery (SSRF) and delivery observability for customers. The signing and
> header format follows the Standard Webhooks specification v1.0.0; error
> bodies follow RFC 9457. It follows the structure of the System Design
> Document template and is pre-filled with a worked design to adapt.
> Replace every `[bracketed]` value and every row marked *Example*; all
> numbers are illustrative assumptions, not benchmarks.

<!--
Guidance: How to use this template
- Remember who owns the other end. Customer endpoints are slow, flaky,
  misconfigured and sometimes hostile; the design is mostly about not
  letting one of them hurt the others or your own network (7.1, 7.4).
- State the delivery contract plainly in customer documentation: at least
  once, possibly out of order, signed, retried for a stated period. Every
  ambiguity becomes a support ticket.
- Size for the failure case. The backlog created when a large customer's
  endpoint is down for a day is the number that sizes storage (3.4).
- Treat every customer URL as untrusted input to your egress network.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-WH-001] |
| System / feature | [Outbound webhooks] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Signature format | [Standard Webhooks v1.0.0, symmetric (`v1`) / asymmetric (`v1a`)] |
| Related | Notification Service System Design Template (`system-design/design-notification-service`), Message Queue System Design Template (`system-design/design-message-queue`), API Design Guidelines Template (`architecture/api-design-guidelines`) |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead, developer platform | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security (egress and SSRF) | [@name] | [Decision] | YYYY-MM-DD |
| Developer experience / docs | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Replaced one shared delivery queue with per-endpoint scheduling |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Moved all egress through a filtering proxy after SSRF review |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Webhooks are sent from the API servers in the request thread
pool through one shared queue. When a large customer's endpoint started
timing out after 30 s last month, worker threads filled up and deliveries
to every other customer were delayed by up to 3 hours. Retries are a fixed
3 attempts over 10 minutes, so short customer outages lose events. Payloads
are unsigned, and a penetration test showed that a registered URL pointing
at the cloud metadata address returned internal credentials.]

### 1.2 Proposed solution

[*Example:* Product services publish domain events to an event log. A
fan-out stage matches each event to subscribed endpoints and creates one
delivery per endpoint. Deliveries are scheduled per endpoint, with a
concurrency cap and a circuit breaker for each endpoint, so one slow
endpoint only delays its own deliveries. Workers send signed HTTPS POST
requests in the Standard Webhooks format through an egress proxy that
blocks private and internal addresses. Failed deliveries are retried with
exponential backoff and jitter for about 3 days, then marked failed and
available for manual or API replay for 30 days. Customers see every attempt
in a delivery log and are notified when an endpoint keeps failing.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* One shared queue or per-endpoint scheduling? | Per-endpoint virtual queues with a fair scheduler | 7.1 |
| Q-02 | *Example:* Do we guarantee ordering? | No by default; optional ordered mode per endpoint with per-resource FIFO | 7.3 |
| Q-03 | *Example:* Symmetric or asymmetric signatures? | Symmetric HMAC (`v1`) by default; asymmetric (`v1a`) as an option | 7.4 |
| Q-04 | *Example:* How long do we retry? | About 3 days, then failed and replayable for 30 days | 7.2 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: Use this order in an interview or a time-boxed review. Spend the most time on one deep dive rather than touching all of them. -->

| Minutes | Cover | Key point to land |
|---|---|---|
| 0 to 5 | Scope: outbound only, customer-registered URLs, event types | The other end is untrusted and unreliable |
| 5 to 10 | Contract: at least once, unordered by default, signed, retried for N days | Receivers must deduplicate by event ID |
| 10 to 15 | Estimates: events per second, fan-out, in-flight requests, failure backlog | Slow endpoints, not volume, size the worker pool |
| 15 to 20 | Data model and access patterns: event, endpoint, delivery, attempt | Due-time index per endpoint |
| 20 to 25 | High-level diagram: event log, fan-out, scheduler, workers, egress proxy | Isolation between endpoints |
| 25 to 38 | One deep dive: per-endpoint isolation with retries and circuit breaking (7.1, 7.2) is the strongest; ordering (7.3) or signing and SSRF (7.4) are good alternatives | One bad endpoint cannot delay others |
| 38 to 42 | Failure modes and customer-facing observability | Replay closes the loop |
| 42 to 45 | Trade-offs and what changes at 10x | Where the design breaks first |

## 2. Requirements

<!-- Guidance: Separate core requirements (the design must meet them) from below-the-line items (explicitly out of scope). The delivery contract belongs here in plain words. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Customers register HTTPS endpoints and choose event types per endpoint | Must |
| FR-02 | *Example:* Deliver each matching event to each subscribed endpoint at least once | Must |
| FR-03 | *Example:* Sign every request; support secret rotation with overlap | Must |
| FR-04 | *Example:* Retry failed deliveries with exponential backoff and jitter for about 3 days | Must |
| FR-05 | *Example:* Show every delivery attempt (status, latency, response excerpt) to the customer | Must |
| FR-06 | *Example:* Replay a single event or a time range to an endpoint | Must |
| FR-07 | *Example:* Verify endpoint ownership before activation | Should |
| FR-08 | *Example:* Optional ordered delivery per resource for endpoints that opt in | Should |
| FR-09 | *Example:* Send test events from the dashboard and API | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Delivery latency | *Example:* p99 below 5 s from event commit to first attempt for healthy endpoints | Pipeline timestamps |
| NFR-02 | Durability | *Example:* No accepted event is lost before it is delivered or expires | Reconciliation job |
| NFR-03 | Isolation | *Example:* One endpoint timing out does not raise p99 first-attempt latency for others by more than 10% | Fault injection |
| NFR-04 | Throughput | *Example:* 105,000 deliveries/s at peak | Load test |
| NFR-05 | Security | *Example:* No request reaches a private, loopback, link-local or internal address | Egress proxy tests |
| NFR-06 | Customer visibility | *Example:* Attempt visible in the dashboard within 10 s | Dashboard probe |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* inbound webhooks from third parties into our product]
- [*Example:* exactly-once delivery (impossible over HTTP without receiver cooperation; receivers deduplicate by event ID)]
- [*Example:* push notifications to devices, email and SMS (Notification Service System Design Template, `system-design/design-notification-service`)]
- [*Example:* event streaming to customer cloud accounts (separate connectors)]
- [*Example:* transforming payloads per customer]

## 3. Capacity estimation

<!-- Guidance: Every estimate must end in a design consequence. Size the failure case as well as the steady state. -->

### 3.1 Assumptions

| Input | Value | Source |
|---|---|---|
| Events per day | *Example:* 2 billion | Event log metrics |
| Peak to average ratio | *Example:* 3 | Hourly profile |
| Endpoints subscribed per event, average | *Example:* 1.5 | Subscription table |
| Registered endpoints | *Example:* 500,000 | Endpoint table |
| Share of endpoints failing at any time | *Example:* 2% | Delivery history |
| Payload size, average | *Example:* 4 KB | Event schemas |
| Healthy endpoint response time | *Example:* 300 ms | Attempt records |
| Request timeout | *Example:* 10 s | Policy (Standard Webhooks suggests 15 to 30 s; shorter here to bound worker time) |
| Replay retention | *Example:* 30 days | Product decision |

### 3.2 Delivery rate

2 billion / 86,400 s = about 23,150 events/s on average, about 69,400/s at
peak (round to 70,000). With 1.5 endpoints per event: about 105,000
deliveries/s at peak.

**Consequence:** a single queue partition or a single scheduler cannot keep
up; partition the scheduler by endpoint ID (*Example:* 256 partitions, about
410 deliveries/s each at peak).

### 3.3 Concurrent requests

By Little's law, healthy traffic needs 105,000/s x 0.3 s = 31,500 requests
in flight. If 1% of deliveries hit slow endpoints that run to the 10 s
timeout, that adds 1,050/s x 10 s = 10,500. Total about 42,000.

**Consequence:** slow endpoints add a third of the in-flight load from 1%
of traffic. Use non-blocking HTTP clients (*Example:* 2,000 in-flight
requests per worker, so 21 workers at peak, plan 32 for headroom and zone
loss) and cap in-flight requests per endpoint so a slow endpoint cannot
take more than its share.

### 3.4 Failure backlog

Deliveries per day = 2 billion x 1.5 = 3 billion. If 2% go to failing
endpoints, 60 million deliveries per day enter retry; held for up to 3 days
that is up to 180 million pending deliveries.

**Consequence:** the pending set does not fit in a broker's memory and must
not block healthy traffic. Store pending deliveries durably with an index on
(`endpoint_id`, `next_attempt_at`) and let a circuit breaker stop attempts
to dead endpoints (7.2) instead of retrying each delivery independently.

### 3.5 Storage

Payloads are stored once per event: 2 billion x 4 KB = 8 TB per day, 240 TB
for 30 days before compression. Attempt records at *Example:* 200 bytes and
1.2 attempts per delivery: 3 billion x 1.2 x 200 B = 720 GB per day, about
21.6 TB for 30 days.

**Consequence:** keep payloads in compressed object storage keyed by event
ID, and only metadata and attempt records in the database; replay reads the
payload from object storage.

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Deliveries at peak | 105,000/s | Scheduler partitioned by endpoint |
| In-flight requests | about 42,000 | Non-blocking workers; per-endpoint caps |
| Pending retries | up to 180 million | Durable due-time index; circuit breakers |
| Payload storage | 240 TB raw for 30 days | Object storage, compressed |
| Attempt records | about 21.6 TB for 30 days | Partitioned table with TTL |

## 4. API design

<!-- Guidance: Define the customer-facing management API, the wire format of the webhook request itself, and the response handling rules. The wire format is a public contract; version it. -->

### 4.1 Management API

| Operation | Purpose |
|---|---|
| `POST /v1/webhook_endpoints` | Register an endpoint URL with event types; returns the signing secret once |
| `PATCH /v1/webhook_endpoints/{id}` | Change URL, event types, ordered mode, or disable |
| `POST /v1/webhook_endpoints/{id}/rotate_secret` | New secret; old one stays valid for an overlap period (*Example:* 24 h) |
| `GET /v1/webhook_endpoints/{id}/deliveries?status=failed` | List deliveries and attempts |
| `POST /v1/webhook_endpoints/{id}/replay` | Replay by event IDs or a time range |
| `POST /v1/webhook_endpoints/{id}/test` | Send a test event |

Management API errors use Problem Details (RFC 9457); for example a
rejected URL returns 422 with the reason (`private address`, `not HTTPS`).

### 4.2 Webhook request format

The request follows Standard Webhooks v1.0.0: three headers and a JSON body
with `type`, `timestamp` and `data`.

```http
POST /hooks/acme HTTP/1.1
Host: hooks.customer.example
Content-Type: application/json
webhook-id: evt_2Lq8x7Kp
webhook-timestamp: 1792656000
webhook-signature: v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDN/H/pI4=

{"type": "invoice.paid", "timestamp": "2026-10-22T08:00:00Z", "data": {"id": "inv_93Fa", "amount": 12000, "currency": "EUR", "status": "paid"}}
```

- `webhook-id` is the event ID and stays the same on every retry, so the
  receiver can use it as an idempotency key.
- `webhook-timestamp` is set per attempt, in Unix seconds.
- `webhook-signature` is `v1,` plus the base64 HMAC-SHA256 of
  `webhook-id.webhook-timestamp.body` using the endpoint secret. During
  rotation it carries two signatures separated by a space.
- The signature value above is illustrative, not a real computation.

Keep payloads small (the specification suggests staying well under about
20 KB); for large objects send an ID and let the receiver fetch the
resource through the API.

### 4.3 Response handling

| Response | Treatment |
|---|---|
| 2xx | Success; delivery complete |
| 3xx | Failure; redirects are not followed (prevents SSRF through redirects); customer is told to update the URL |
| 410 Gone | Disable the endpoint and notify the customer |
| 429, 502, 504 | Failure; slow down this endpoint; honour `Retry-After` if present (the throttling codes named by Standard Webhooks) |
| 503 | Failure; this design also slows down and honours `Retry-After` (a local choice: the specification treats 503 as an ordinary failure) |
| Other 4xx and 5xx | Failure; retry per schedule |
| Timeout, TLS or connection error | Failure; retry per schedule |

## 5. Data model

<!-- Guidance: Separate the immutable event from the per-endpoint delivery and its attempts. Replay creates new attempts, never edits old ones. -->

### 5.1 Entities

One event fans out to many deliveries; each delivery has one or more
attempts.

```mermaid
erDiagram
  ENDPOINT {
    string endpoint_id PK
    string account_id
    string url
    string event_types
    bool ordered
    string state
    string secret_refs
  }
  EVENT {
    string event_id PK
    string account_id
    string type
    datetime created_at
    string payload_ref
  }
  DELIVERY {
    string delivery_id PK
    string event_id FK
    string endpoint_id FK
    string ordering_key
    string status
    int attempt_count
    datetime next_attempt_at
  }
  ATTEMPT {
    string delivery_id PK
    int attempt_no PK
    datetime started_at
    int status_code
    int latency_ms
    string error_class
    string response_excerpt
  }
  ENDPOINT_HEALTH {
    string endpoint_id PK
    string breaker_state
    float failure_rate
    datetime open_until
  }
  ENDPOINT ||--o{ DELIVERY : receives
  EVENT ||--o{ DELIVERY : "fans out to"
  DELIVERY ||--|{ ATTEMPT : "tried as"
  ENDPOINT ||--|| ENDPOINT_HEALTH : "tracked by"
```

### 5.2 Storage choices

| Data | Store | Why |
|---|---|---|
| Events (metadata) | Event log plus partitioned table | Fan-out source and replay index |
| Payloads | Object storage, compressed | 240 TB per 30 days at low cost |
| Deliveries and due times | *Example:* Partitioned database by `endpoint_id` | Durable pending set with due-time index |
| Attempts | Partitioned table with 30-day TTL | Customer delivery log |
| Endpoint config and secrets | Relational database; secrets in a key management service, referenced | Strong consistency, audit, encryption |
| Endpoint health | In memory on scheduler partition, checkpointed | Breaker decisions per attempt |

### 5.3 Access patterns

| Query | Served by | Key or index | Partition rationale |
|---|---|---|---|
| Which endpoints subscribe to event type T for account A | Subscription cache | (`account_id`, `type`) | Small, cached in fan-out workers |
| Next due deliveries for endpoint E | Deliveries table | (`endpoint_id`, `next_attempt_at`) | Partition by `endpoint_id` so one scheduler partition owns an endpoint |
| Endpoints with due work on partition P | Ready index | (`partition`, `earliest_due_at`) | Lets the scheduler skip idle endpoints |
| Delivery log for endpoint E, newest first | Attempts table | (`endpoint_id`, `started_at` desc) | Same partition as deliveries |
| All deliveries of event X (support) | Secondary index | `event_id` | Low volume |
| Replay range for endpoint E | Events table | (`account_id`, `created_at`) filtered by type | Range scan, rate-limited |
| Next item for ordering key K | Deliveries table | (`endpoint_id`, `ordering_key`, `sequence`) | Same partition as the endpoint |

## 6. High-level design

<!-- Guidance: Show the path from a committed domain event to an HTTP request on the customer's server, and where isolation and security controls sit. -->

```mermaid
flowchart LR
  svc[Product services] --> outbox[(Transactional outbox)]
  outbox --> log[(Event log)]
  log --> fan[Fan-out workers]
  fan --> payloads[(Payload store)]
  fan --> dstore[(Deliveries, partitioned by endpoint)]
  subgraph sched["Scheduler partitions"]
    s1[Partition 0..255: per-endpoint queues, breakers, caps]
  end
  dstore --> s1
  s1 --> wk[Delivery workers, non-blocking HTTP]
  wk --> sign[Signer, secrets from KMS]
  wk --> px[Egress proxy: SSRF filter]
  px --> ep([Customer endpoints])
  wk --> att[(Attempt records)]
  att --> dash[Customer dashboard and API]
  s1 --> alert[Endpoint failure notifications]
```

| Component | Responsibility | Scales by |
|---|---|---|
| Transactional outbox | Publish events only when the business transaction commits | Per service |
| Fan-out workers | Match events to endpoints, store payload, create deliveries | Event log partitions |
| Scheduler partitions | Own endpoints; pick due deliveries fairly; apply caps and breakers | 256 partitions |
| Delivery workers | Sign, send, record attempts | In-flight requests |
| Egress proxy | Resolve, validate and connect only to public addresses | Proxy instances |
| Dashboard and API | Delivery log, replay, test events | Read replicas |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. For each deep dive list the options, the choice and the reason. -->

### 7.1 Deep dive 1: per-endpoint isolation

| Option | Isolation | Cost | Verdict |
|---|---|---|---|
| A. One shared FIFO queue | None: a slow endpoint holds workers and blocks the queue | Lowest | The current outage |
| B. Physical queue per endpoint in the broker | Strong | 500,000 queues; broker limits | Too many queues to operate |
| C. Virtual per-endpoint queues in a partitioned store with a fair scheduler | Strong | Index per endpoint | Chosen |

**Choice:** C. **Reason:** it gives each endpoint its own backlog and limits
without creating broker objects per customer. Each scheduler partition:

- Keeps a ready set of endpoints with due deliveries, ordered by earliest
  due time.
- Takes deliveries in rounds across endpoints (deficit round robin), so a
  customer with a million pending events gets the same turn as one with
  ten.
- Enforces a per-endpoint in-flight cap (*Example:* 10 by default, raised
  for large customers on request) and a per-account cap.
- Never assigns more work to an endpoint whose breaker is open.

The delivery sequence for one healthy endpoint:

```mermaid
sequenceDiagram
  participant S as Scheduler partition
  participant W as Delivery worker
  participant K as Key management
  participant P as Egress proxy
  participant E as Customer endpoint
  S->>W: delivery d1 (endpoint e7, attempt 1)
  W->>K: signing secret for e7 (cached)
  W->>W: build body, sign id.timestamp.body
  W->>P: POST to e7 URL (timeout 10 s)
  P->>P: resolve DNS, reject private ranges, pin IP
  P->>E: HTTPS POST
  E-->>P: 200
  P-->>W: 200 in 240 ms
  W->>S: success, record attempt
  S->>S: mark delivered, update health
```

### 7.2 Deep dive 2: retries, backoff and circuit breaking

Retry schedule (*Example*, each delay with plus or minus 20% jitter):

| Attempt | Delay after previous | Cumulative |
|---|---|---|
| 1 | immediate | 0 |
| 2 | 5 s | 5 s |
| 3 | 1 min | about 1 min |
| 4 | 10 min | about 11 min |
| 5 | 1 h | about 1 h 11 min |
| 6 | 3 h | about 4 h 11 min |
| 7 | 6 h | about 10 h 11 min |
| 8 | 12 h | about 22 h 11 min |
| 9 | 24 h | about 46 h 11 min |
| 10 | 24 h | about 70 h 11 min (just under 3 days) |

| Option for dead endpoints | Behaviour | Verdict |
|---|---|---|
| A. Retry each delivery on its own schedule | 180 million pending deliveries each retried on their own | Wastes capacity on known-dead endpoints |
| B. Circuit breaker per endpoint | After failures, stop sending; probe periodically; release backlog on recovery | Chosen |
| C. Disable endpoint after first failures | Fast, but drops events for brief outages | Only after sustained failure |

**Choice:** B, then C after sustained failure. **Reason:** an endpoint is
usually down as a whole, so its deliveries should wait together. The
breaker opens after *Example:* 10 consecutive failures or more than 50%
failures over 5 minutes, sends one probe delivery per backoff interval, and
on success closes and drains the backlog at the endpoint's cap so recovery
does not flood the customer. Deliveries still respect their own expiry:
after about 3 days without success they become `failed` and stay
replayable for 30 days. After *Example:* 5 days of continuous failure the
endpoint is disabled and the customer is notified by email and in the
dashboard, as the specification recommends.

### 7.3 Deep dive 3: ordering

| Option | Guarantee | Cost | Verdict |
|---|---|---|---|
| A. No ordering | None; receivers fetch current state if needed | None | Default |
| B. FIFO per endpoint | Strict order | One failing event blocks all events to that endpoint | Rejected |
| C. FIFO per ordering key (resource ID) within an endpoint, opt-in | Order per resource, parallel across resources | Head-of-line blocking per resource | Opt-in mode |

**Choice:** A by default, C for endpoints that opt in. **Reason:** strict
ordering turns any single failure into a full stop, and most receivers only
need the latest state. Every payload carries the event creation time and a
per-resource sequence number so receivers can discard older updates. In
ordered mode the scheduler releases the next delivery for a key only after
the previous one succeeded or expired, and the dashboard shows blocked keys.

### 7.4 Deep dive 4: signing, endpoint verification and SSRF

Signing (Standard Webhooks v1.0.0):

- One secret per endpoint, 24 to 64 random bytes, shown once with the
  `whsec_` prefix; stored in a key management service.
- Symmetric signatures use HMAC-SHA256 (RFC 2104) with the `v1` prefix;
  asymmetric signatures use ed25519 with the `v1a` prefix for customers who
  must not hold a shared secret.
- Sign the exact bytes sent; never re-serialise the body after signing.
- Rotation sends two signatures during the overlap period.
- Customer documentation tells receivers to compare signatures in constant
  time, reject timestamps outside a tolerance (*Example:* 5 minutes) to
  stop replays, and deduplicate by `webhook-id`.

Endpoint verification: on registration and on URL change, the platform
sends a `webhook.verification` event containing a random challenge; the
endpoint is activated only after a 2xx response that echoes the challenge.
This proves the customer controls the URL and stops the platform from being
used to send traffic to third parties.

SSRF protection (OWASP Server Side Request Forgery Prevention Cheat Sheet):

| Control | Detail |
|---|---|
| HTTPS only | Reject `http://` URLs in production; TLS 1.2 or later |
| Validate at registration and at send time | DNS can change after registration, so check on every connection |
| Resolve, check, then connect to that IP | Block loopback, private, link-local (including cloud metadata), carrier-grade NAT, multicast and IPv6 equivalents; pin the checked address to stop DNS rebinding |
| No redirects | 3xx is a failure (4.3) |
| Egress proxy | All delivery traffic leaves through a filtering proxy (*Example:* an open-source SSRF proxy such as Stripe's smokescreen) in a network segment with no route to internal services |
| Port allow-list | 443 and [8443] only |
| Response limits | Read at most *Example:* 64 KB of the response; store a 1 KB excerpt |

> **How real systems do it** (paraphrased from Stripe's public webhook
> documentation; see 12.4). Stripe tells integrators that live-mode events
> are retried with exponential backoff for up to three days, that delivery
> order is not guaranteed and the same event can arrive more than once, so
> receivers should record processed event IDs. Each request carries a
> signature header with a timestamp and an HMAC-SHA256 signature over the
> timestamp and raw body; the official libraries reject timestamps older
> than a default tolerance of five minutes, and a new signature and
> timestamp are generated for each retry. Redirect responses count as
> failures. Secrets can be rolled with a period during which both old and
> new secrets produce signatures. Receivers are told to return a 2xx
> quickly and do heavy work asynchronously. Lesson for this design: publish
> the same plain contract (at least once, unordered, signed, retried for a
> stated period) and give receivers the tools to follow it.

### 7.5 Customer-facing observability and replay

- Delivery log per endpoint: event ID, type, attempt number, status code,
  latency, error class (timeout, TLS, DNS, refused, HTTP status) and a
  response excerpt.
- Endpoint health page: success rate over 24 hours, p95 latency, breaker
  state, backlog size and the time of the next attempt.
- Replay: by event ID, by filter, or by time range; replays create new
  deliveries flagged `replay=true`, keep the original `webhook-id`, and are
  rate-limited per endpoint.
- Notifications: email and dashboard banner when the breaker opens, again
  before the endpoint is disabled, and a summary when it recovers.
- Events API: customers can list recent events through the API to reconcile
  anything they missed, which is the safety net behind at-least-once
  delivery.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it breaks first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Delivery rate | Scheduler partitioned by endpoint; stateless workers | *Example:* about 1,000 deliveries/s per partition |
| Slow endpoints | Per-endpoint in-flight caps; 10 s timeout | Worker in-flight capacity |
| Big customer bursts | Per-account caps; fair rounds across endpoints | Customer's own endpoint capacity |
| Pending backlog | Durable due-time index; breakers | Storage, not memory |
| Payload storage | Object storage with lifecycle rules | Cost, not technology |
| Egress proxy | Horizontally scaled, connection reuse per destination | Proxy connection table |

Latency budget for the first attempt to a healthy endpoint (target p99 5 s
from event commit):

| Step | Budget |
|---|---|
| Outbox relay to event log | 500 ms |
| Fan-out and delivery creation | 500 ms |
| Scheduler pickup | 1,000 ms |
| Signing and proxy checks | 50 ms |
| Network and customer processing | Customer-dependent (excluded) |
| Headroom | 2,950 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. Each mitigation should be testable in a game day. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Customer endpoint | Down or slow | Breaker state, attempt errors | That endpoint only | Breaker, backoff, notifications, replay | Customer-dependent |
| Scheduler partition | Host crash | Partition lease lost | About 0.4% of endpoints (1 of 256) delayed | Lease moves to another host; state rebuilt from store | *Example:* 30 s |
| Delivery workers | Crash mid-request | Lease timeout on delivery | Some deliveries sent twice | At-least-once contract; receivers deduplicate | *Example:* 30 s |
| Egress proxy | Down | Proxy health, connect errors | All deliveries in the region | Multiple instances per zone; deliveries wait and retry, never bypass the proxy | *Example:* 5 min |
| Key management | Unavailable | Signing errors | All deliveries | Cache data keys in memory for a bounded time; deliveries wait if expired | *Example:* 15 min |
| Event log | Unavailable | Outbox relay lag | New events delayed | Outbox holds events in each service's database | Log RTO |
| Fan-out bug | Events sent to wrong endpoint | Account mismatch check before send | Potential data leak across customers | Pre-send assertion that delivery account equals event account; kill switch | *Example:* immediate stop |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [RTO] |

### 9.2 Security

| Threat | Control |
|---|---|
| SSRF to internal services or metadata | Egress proxy, address checks at send time, no redirects (7.4) |
| Spoofed webhooks to customers | Signatures; documented verification; asymmetric option |
| Replay of captured requests | Timestamp in the signed content; receiver tolerance window |
| Secret leakage | Secrets in KMS; shown once; rotation API; secrets never logged |
| Data sent to the wrong customer | Account assertion before send; endpoint verification |
| Sensitive data in payloads | Minimal payloads with IDs; customers fetch details through the authenticated API |
| Abuse as a traffic amplifier | Endpoint verification; per-account limits |
| [Threat] | [Control] |

Run a threat model (Threat Model Template, `security/threat-model`) on the
egress path and the fan-out account check before launch.

### 9.3 Observability (operator)

| Signal | Source | Alert threshold |
|---|---|---|
| First-attempt latency p99 | Pipeline timestamps | *Example:* above 5 s for 10 min |
| Fan-out lag | Event log consumer lag | *Example:* above 60 s |
| Open breakers | Scheduler | Sudden rise across many endpoints (suggests our egress, not customers) |
| Proxy rejections by reason | Egress proxy | Any spike in private-address rejections |
| Success rate excluding open breakers | Attempts | *Example:* below 98% |
| Pending backlog | Deliveries table | Growth beyond [n] per hour |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Delivery workers and schedulers | [USD per instance-month] | *Example:* 32 + 16 | [USD] |
| Egress proxy and NAT | [USD per GB] | [GB per month] | [USD] |
| Delivery and attempt storage | [USD per TB-month] | *Example:* about 22 TB | [USD] |
| Payload object storage | [USD per TB-month] | *Example:* 240 TB raw, less after compression | [USD] |
| **Total** | | | **[USD]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Include how the design changes at ten times and one tenth of the load. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Guarantee | At most once; at least once | At least once | No silent loss | Receivers must deduplicate |
| Ordering | None; per endpoint; per key opt-in | None by default, per key opt-in | Avoids head-of-line blocking | Receivers handle out-of-order events |
| Isolation | Shared queue; broker queue per endpoint; virtual queues | Virtual queues | Strong isolation without broker sprawl | Custom scheduler to build and run |
| Payload | Full object; thin (ID only) | Full object, small, with option for thin | Fewer API calls for receivers | Data duplication; stale snapshots |
| Signature | Symmetric; asymmetric | Symmetric default, asymmetric option | Simple for most receivers | Shared secret to protect |
| Build or buy | Build; webhook-as-a-service vendor | [Decide] | [Reason] | [Cost] |
| What changes at 10x | 1 million deliveries/s, 5 million endpoints | More scheduler partitions, regional delivery fleets near customers, tiered payload storage | Scheduler partition throughput and egress IP reputation bind first | More regions to operate; customers must allow more source IPs |
| What changes at 0.1x | 10,500 deliveries/s | A single queue table with per-endpoint locking and a managed queue may suffice; or use a vendor | Team cost dominates | Weaker isolation unless caps are kept |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Egress proxy and SSRF controls on the existing sender | Penetration retest passes | YYYY-MM-DD |
| M2 | New pipeline in shadow mode (deliver to a sink, compare) | Same deliveries as legacy for 7 days | YYYY-MM-DD |
| M3 | Signed deliveries, per-endpoint scheduler, delivery log | Isolation fault test passes | YYYY-MM-DD |
| M4 | Replay, ordered mode, endpoint verification | Customer beta feedback addressed | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Move accounts in cohorts from the legacy sender to the new
pipeline. Signatures are added before the cutover with at least 60 days'
notice so customers can enable verification. Rollback routes a cohort back
to the legacy sender; deliveries already created in the new pipeline
complete there.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Customers do not verify signatures | High | Medium | SDK helpers, docs, verification tool, test events | [@name] |
| R-02 | *Example:* Duplicate deliveries during migration | Medium | Medium | Keep the same event IDs; announce at-least-once contract | [@name] |
| R-03 | *Example:* Egress IP changes break customer allow-lists | Medium | High | Publish IP list with 30 days' notice of changes | [@name] |
| R-04 | *Example:* Ordered mode causes long stalls | Medium | Medium | Dashboard shows blocked keys; expiry releases them | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary, open questions and references. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Offer thin events (ID only) as the default for new endpoints? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Should enterprise customers get dedicated egress IPs? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Traceability

| Requirement | Design section | Verification |
|---|---|---|
| FR-02, NFR-02 | 7.1, 9.1 | Reconciliation job comparing events and deliveries |
| FR-03 | 7.4 | Signature conformance with Standard Webhooks reference libraries |
| FR-04 | 7.2 | Endpoint outage simulation of 2 days, then recovery drain |
| FR-06 | 7.5 | Replay tests by ID and by range |
| NFR-03 | 7.1 | One endpoint slowed to timeout; other endpoints unchanged |
| NFR-05 | 7.4 | SSRF test suite: private ranges, DNS rebinding, redirects, IPv6 |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Endpoint | A customer URL registered to receive events |
| Delivery | One event destined for one endpoint |
| Attempt | One HTTP request for a delivery |
| Circuit breaker | Per-endpoint switch that stops attempts after repeated failures and probes for recovery |
| Ordering key | Resource identifier used for optional per-resource FIFO |
| SSRF | Server-side request forgery: tricking a server into calling an address it should not reach |
| Replay | Re-sending past events to an endpoint on request |

### 12.4 References

| Reference | Use |
|---|---|
| Standard Webhooks specification v1.0.0 (github.com/standard-webhooks/standard-webhooks, Apache 2.0) | Headers, signed content, secret format, status handling, retry guidance (4.2, 4.3, 7.4) |
| Stripe documentation, "Receive Stripe events in your webhook endpoint" (docs.stripe.com/webhooks) | Inspiration for the box in 7.4 (paraphrased) |
| OWASP Server Side Request Forgery Prevention Cheat Sheet | SSRF controls in 7.4 |
| Stripe smokescreen (github.com/stripe/smokescreen) | Example egress proxy |
| RFC 2104 HMAC; RFC 9110 HTTP Semantics; RFC 9457 Problem Details | Signing primitive, status codes, management API errors |
| Notification Service System Design Template (`system-design/design-notification-service`) | Device, email and SMS notifications |
| Message Queue System Design Template (`system-design/design-message-queue`) | Event log semantics |
| Rate Limiter System Design Template (`system-design/design-rate-limiter`) | Per-account caps |
| Resilience and Load Shedding Design Template (`system-design/resilience-load-shedding-design`) | Circuit breaker policy |

## Standard and licence note

Based on the Standard Webhooks specification v1.0.0 for the request format
and signatures, on the OWASP SSRF Prevention Cheat Sheet for egress
controls, and on RFC 2104, RFC 9110 and RFC 9457. Stripe material is used
as inspiration only and is paraphrased, not copied. All numbers are
illustrative examples. Check the current version of each specification
before you implement. Original text written for this collection. Template
licence: MIT.