# Event Delivery Guarantees: [Flow or service]

> Building-block decision record for how messages and events travel between
> services without being lost or applied twice. It covers the delivery
> semantics per flow (at-most-once, at-least-once, effectively-once),
> publishing atomically with a database change (transactional outbox),
> idempotent consumers, ordering, retries and dead-letter handling,
> multi-step business transactions as sagas, and the limits of
> "exactly-once" claims. Concepts draw on the transactional outbox pattern
> as described on microservices.io, on sagas (Garcia-Molina and Salem,
> SIGMOD 1987), on Helland's "Life Beyond Distributed Transactions" (ACM
> Queue, 2016) and on the delivery semantics section of the Apache Kafka
> 4.4 documentation. Replace every `[bracketed]` value and every row marked
> *Example*. Worked examples use Quillmere, an invented book-review and
> reading-list platform shared by the other building-block templates in
> this category.

<!--
Guidance: How to use this template
- Assume at-least-once delivery everywhere and design every consumer to
  be idempotent. Exactly-once processing exists only inside boundaries
  that control both the input position and the output (section 6).
- Fill the flow register (section 1) first: one row per producer and
  consumer pair, with the business effect of a lost message and of a
  duplicate. That pair of answers decides the semantics.
- Never publish to a broker inside a database transaction or right after
  commit without an outbox; both lose or invent events on a crash.
- For the event model itself (event sourcing, read models) use the Event
  Sourcing and CQRS Design template (`architecture/event-sourcing-cqrs`);
  this record covers the delivery contract between services.
-->

## Document control

<!-- Guidance: Producer and consumer teams both sign; the contract binds both sides. -->

| Field | Value |
|---|---|
| Document ID | [EVT-REC-001] |
| Flows covered | [*Example:* Review posted, rating changed, reading goal completed, gift purchase saga] |
| Broker or transport | [*Example:* Partitioned log broker, topics per event type] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@platform-events-lead] |
| Producer teams | [@reviews-lead], [@orders-lead] |
| Consumer teams | [@notifications-lead], [@recs-lead] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Related | Event Sourcing and CQRS Design (`architecture/event-sourcing-cqrs`), Message Queue System Design (`system-design/design-message-queue`), CDC Pipeline System Design (`system-design/design-cdc-pipeline`) |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Platform events | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Producer team | [@name] | [Decision] | YYYY-MM-DD |
| Consumer team | [@name] | [Decision] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added outbox for review events after lost-notification incident |

## 1. Flow register

<!-- Guidance: One row per producer to consumer flow. The two impact columns are the most important ones: they show which semantics you can afford. -->

| ID | Event | Producer | Consumer | Impact if lost | Impact if duplicated | Semantics chosen |
|---|---|---|---|---|---|---|
| EV-01 | *Example:* `review.posted` | Reviews | Notifications | Author's followers not told | Two notifications | At-least-once, dedup by event ID |
| EV-02 | *Example:* `rating.changed` | Reviews | Recommendations | Slightly stale model input | Counted twice in a feature | At-least-once, idempotent upsert |
| EV-03 | *Example:* `page.viewed` | Web | Analytics | One view uncounted | One view overcounted | At-most-once acceptable |
| EV-04 | *Example:* `gift.payment.captured` | Payments | Orders | Customer charged, no gift sent | Gift sent twice | At-least-once, idempotent by payment ID, saga (section 7) |
| [EV-NN] | [Event] | [Service] | [Service] | [Impact] | [Impact] | [Semantics] |

## 2. Delivery semantics

<!-- Guidance: Use precise terms. The broker can promise how often a message is delivered; only the consumer can make processing effectively once. -->

| Semantics | Meaning | How it arises | Use when |
|---|---|---|---|
| At-most-once | Each message delivered zero or one time | Acknowledge before processing, no retries | Loss is cheap (telemetry, some analytics) |
| At-least-once | Each message delivered one or more times | Acknowledge after processing, retry on doubt | Default for business events |
| Effectively-once (exactly-once processing) | The effect of each message is applied once, even if delivered more than once | At-least-once delivery plus an idempotent consumer, or a transactional read-process-write inside one system | Money, inventory, anything counted |

The Apache Kafka documentation (4.4) explains why: when a producer gets
no reply it cannot know whether the write was committed, so resending
gives at-least-once behaviour. The idempotent producer removes duplicates
caused by producer retries within the log; transactions let a
read-process-write application commit its output and its input offsets
together, and consumers in `read_committed` isolation see only committed
data. The same documentation notes that writing to an external system
needs that system's cooperation to coordinate output with the consumer
position.

## 3. Publishing: transactional outbox

<!-- Guidance: The producer must not lose an event when the business change commits, and must not emit an event for a change that rolled back. The outbox makes both one database transaction; a relay then publishes. Expect the relay to publish duplicates after a crash. -->

```mermaid
sequenceDiagram
  participant S as Reviews service
  participant DB as Reviews database
  participant R as Outbox relay
  participant B as Broker
  participant C as Notifications consumer
  S->>DB: BEGIN, insert review, insert outbox row, COMMIT
  R->>DB: Read unpublished outbox rows in order
  R->>B: Publish review.posted (event ID e-981)
  B-->>R: Acknowledged
  R->>DB: Mark e-981 published
  B->>C: Deliver e-981
  C->>C: Skip if e-981 already processed, else apply and record
```

| Decision | Options | *Example* choice |
|---|---|---|
| Relay type | Polling publisher, or transaction log tailing (change data capture) | Log tailing via the CDC pipeline (`system-design/design-cdc-pipeline`) |
| Event ID | UUID or database sequence, assigned in the transaction | UUIDv7 assigned by the service |
| Ordering | Per aggregate: partition key = aggregate ID | Partition key `book_id` |
| Outbox cleanup | Delete or archive published rows after a retention period | Delete after 3 days |
| Volume | Average and peak events per second | 20,000,000 per day, about 231 per second on average |

## 4. Idempotent consumers

<!-- Guidance: Every consumer of an at-least-once flow must state how it detects or tolerates duplicates. Natural idempotency (set a value, upsert by key) is cheaper than a dedup table; use the table only when the effect is additive. -->

| Technique | How it works | Fits |
|---|---|---|
| Natural idempotency | The operation gives the same result when repeated (set status, upsert by key) | State updates |
| Dedup table | Record processed event IDs in the same transaction as the effect; skip known IDs | Additive effects (send, count, charge) |
| Version check | Apply only if the event's version is newer than the stored one | Out-of-order updates to one entity |
| Idempotency key at the target | Pass a key to the downstream API so it rejects repeats | Payments and external APIs |

Dedup table sizing, *Example:* 20,000,000 events per day kept for 7 days
is 140,000,000 rows; at about 64 bytes per row including index overhead
that is 8.96 GB. The retention must exceed the longest redelivery window
(broker retention plus replay periods), or a late duplicate is applied
again.

## 5. Ordering, retries and dead letters

<!-- Guidance: Decide per flow whether order matters and within what scope. Global order is rarely needed and is expensive; order per aggregate is usually enough. A retry that blocks a partition keeps order but stalls every later event for that partition. -->

| Rule | *Example* decision |
|---|---|
| Ordering scope | Per `book_id` (one partition per key) |
| Retry policy | Exponential backoff 1, 2, 4, 8, 16, 32 s then 60 s cap; first attempt plus 8 retries, 183 s of waiting in total |
| Poison message | After the last retry, move to a dead-letter topic with error, attempt count and original offset |
| Ordering versus retry | Retrying event moves to a retry topic; later events for the same key wait behind it (key is parked) |
| Dead-letter handling | Alert on any message; owning team triages within 1 business day; replay tool re-injects after fix |
| Schema evolution | Consumers ignore unknown fields; breaking changes use a new event type or version |

Backoff schedule, *Example:*

| Retry | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| Wait before retry (s) | 1 | 2 | 4 | 8 | 16 | 32 | 60 | 60 |
| Cumulative wait (s) | 1 | 3 | 7 | 15 | 31 | 63 | 123 | 183 |

Add random jitter to each wait in production so that many consumers do
not retry in step; see the Resilience and Load Shedding Design template
(`system-design/resilience-load-shedding-design`).

## 6. Exactly-once: limits

<!-- Guidance: Record exactly what the "exactly-once" in your stack covers. Any side effect outside the transactional boundary (email, HTTP call, another database) is at-least-once again unless that system takes an idempotency key. -->

| Boundary | Effectively once? | Why |
|---|---|---|
| Broker topic to broker topic with transactional read-process-write | Yes, within the broker | Output and input offsets committed together |
| Broker to consumer database, offset stored in the same database transaction | Yes | Effect and position committed atomically |
| Broker to consumer database, offset stored in the broker | No, at-least-once | Crash between database commit and offset commit replays |
| Consumer sends email or push notification | No, at-least-once | External system cannot join the transaction; dedup by event ID before sending narrows but does not close the gap |
| Consumer calls a payment API with an idempotency key | Yes, if the API honours the key for the retry window | Target rejects repeats |

## 7. Sagas for multi-service transactions

<!-- Guidance: When one business action spans services, use a saga: a sequence of local transactions, each with a compensating action. Choose orchestration (a coordinator tells each step what to do) or choreography (each service reacts to events). Compensations must themselves be idempotent and must be possible: some steps cannot be undone and must come last. -->

*Example:* gift purchase saga (orchestrated).

| Step | Service | Action | Compensation | Can it be undone? |
|---|---|---|---|---|
| 1 | Orders | Create gift order (pending) | Cancel order | Yes |
| 2 | Payments | Authorise card | Void authorisation | Yes |
| 3 | Inventory | Reserve e-book licence | Release licence | Yes |
| 4 | Payments | Capture payment | Refund | Yes, with fees |
| 5 | Notifications | Email gift code to recipient | None (cannot unsend) | No, so it runs last |

```mermaid
stateDiagram-v2
  [*] --> OrderPending
  OrderPending --> Authorised: payment authorised
  Authorised --> Reserved: licence reserved
  Reserved --> Captured: payment captured
  Captured --> Delivered: gift email sent
  Delivered --> [*]
  Authorised --> Voided: reservation failed
  Voided --> Cancelled: order cancelled
  Cancelled --> [*]
```

For long-running orchestration with timers and durable state, see the
Workflow Orchestration Engine System Design template
(`system-design/design-workflow-orchestration-engine`).

## 8. Monitoring

<!-- Guidance: Measure loss and duplication directly, not only broker health. -->

| Signal | *Example* alert |
|---|---|
| Outbox backlog (unpublished rows, oldest age) | Oldest unpublished older than 60 s |
| Consumer lag per group and partition | Lag above 5 minutes |
| Dead-letter count | Any new message |
| Duplicate rate (dedup table hits per second) | Above 1% of deliveries for 15 minutes |
| End-to-end reconciliation | Daily count of `review.posted` events vs new reviews differs by more than 0 |
| Saga stuck in a non-final state | Older than 30 minutes |

## 9. Decisions and open items

<!-- Guidance: Record each decision with its reason; link larger ones to an ADR (`architecture/adr-full`). -->

| ID | Decision or open item | Reason | Owner | Status |
|---|---|---|---|---|
| ED-01 | *Example:* All business events via transactional outbox | No lost or phantom events on crash | @platform-events-lead | Decided |
| ED-02 | *Example:* Consumers keep a 7-day dedup table for additive effects | Broker retention is 3 days, replays within 7 | @notifications-lead | Decided |
| ED-03 | *Example:* Gift purchase as orchestrated saga, email last | Email cannot be compensated | @orders-lead | Decided |
| ED-04 | *Example:* Decide whether analytics may drop events under load | At-most-once acceptable for page views | @analytics-lead | Open |
| [ED-NN] | [Item] | [Reason] | [@owner] | [Status] |

## Glossary and references

<!-- Guidance: Keep terms consistent with section 2. -->

| Term | Meaning |
|---|---|
| Outbox | Table written in the same transaction as the business change, holding events to publish |
| Relay | Process that reads the outbox and publishes to the broker |
| Idempotent | Applying an operation more than once has the same effect as once |
| Dead-letter topic | Destination for messages that failed all retries |
| Saga | Sequence of local transactions with compensating actions instead of one distributed transaction |
| Compensation | Action that semantically undoes an earlier step |

References:

- Richardson C., "Pattern: Transactional outbox", microservices.io,
  https://microservices.io/patterns/data/transactional-outbox.html
  (checked 2026-10-30).
- Garcia-Molina H., Salem K., "Sagas", Proceedings of the 1987 ACM SIGMOD
  International Conference on Management of Data, pages 249 to 259,
  https://doi.org/10.1145/38713.38742.
- Helland P., "Life Beyond Distributed Transactions", ACM
  Queue 14(5), 2016, pages 69 to 98, https://doi.org/10.1145/3012426.3025012.
- Apache Kafka 4.4 documentation, Design, delivery semantics,
  https://kafka.apache.org/44/design/design/, and producer configuration
  `enable.idempotence`, https://kafka.apache.org/44/configuration/producer-configs/
  (checked 2026-10-30).
- Related templates: Event Sourcing and CQRS Design
  (`architecture/event-sourcing-cqrs`), Event-Driven Architecture Design
  (`architecture/event-driven-architecture-design`), Message Queue System
  Design (`system-design/design-message-queue`), Webhook Delivery System
  Design (`system-design/design-webhook-delivery`), Payment Ledger System
  Design (`system-design/design-payment-ledger`).

## Standard and licence note

Based on the publicly available papers, the Apache Kafka documentation and
the pattern description cited above; no standard governs event delivery
between services. The text, tables, numbers and examples are original to
this collection, and Quillmere is an invented organisation. Broker
features differ and change between versions: check the current
documentation of the broker and client libraries you use. Template
licence: MIT.