Event Delivery Guarantees Template (Outbox, Idempotency, Sagas)

Flow register, delivery semantics, transactional outbox, idempotent consumers, retries and dead letters, exactly-once limits, sagas

SystemdesignStandardTransactional outbox; Sagas (Garcia-Molina and Salem 1987); Apache Kafka 4.4 delivery semanticsevent delivery guaranteesAktualisiert 29.10.2026

Im Editor öffnen.md herunterladen

Anleitung: Use it when services exchange events or messages that must not be lost or applied twice. Fill the flow register with the impact of a lost and a duplicated message first. The main pitfall is trusting an exactly-once claim beyond the boundary that commits output and input position together.

Vorschau

Event Delivery Guarantees: [Flow or service]

Building-block decision record for how messages and events travel between services without being lost or applied twice. It covers the delivery semantics per flow (at-most-once, at-least-once, effectively-once), publishing atomically with a database change (transactional outbox), idempotent consumers, ordering, retries and dead-letter handling, multi-step business transactions as sagas, and the limits of "exactly-once" claims. Concepts draw on the transactional outbox pattern as described on microservices.io, on sagas (Garcia-Molina and Salem, SIGMOD 1987), on Helland's "Life Beyond Distributed Transactions" (ACM Queue, 2016) and on the delivery semantics section of the Apache Kafka 4.4 documentation. Replace every [bracketed] value and every row marked Example. Worked examples use Quillmere, an invented book-review and reading-list platform shared by the other building-block templates in this category.

Document control

Field	Value
Document ID	[EVT-REC-001]
Flows covered	[Example: Review posted, rating changed, reading goal completed, gift purchase saga]
Broker or transport	[Example: Partitioned log broker, topics per event type]
Version	[0.1]
Status	[Draft / In review / Approved / Implemented / Superseded]
Owner	[@platform-events-lead]
Producer teams	[@reviews-lead], [@orders-lead]
Consumer teams	[@notifications-lead], [@recs-lead]
Last updated	YYYY-MM-DD
Classification	[Internal]
Related	Event Sourcing and CQRS Design (`architecture/event-sourcing-cqrs`), Message Queue System Design (`system-design/design-message-queue`), CDC Pipeline System Design (`system-design/design-cdc-pipeline`)

Approvals

Role	Name	Decision	Date
Platform events	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Producer team	[@name]	[Decision]	YYYY-MM-DD
Consumer team	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft
0.2	YYYY-MM-DD	[@name]	Example: Added outbox for review events after lost-notification incident

1. Flow register

ID	Event	Producer	Consumer	Impact if lost	Impact if duplicated	Semantics chosen
EV-01	Example: `review.posted`	Reviews	Notifications	Author's followers not told	Two notifications	At-least-once, dedup by event ID
EV-02	Example: `rating.changed`	Reviews	Recommendations	Slightly stale model input	Counted twice in a feature	At-least-once, idempotent upsert
EV-03	Example: `page.viewed`	Web	Analytics	One view uncounted	One view overcounted	At-most-once acceptable
EV-04	Example: `gift.payment.captured`	Payments	Orders	Customer charged, no gift sent	Gift sent twice	At-least-once, idempotent by payment ID, saga (section 7)
[EV-NN]	[Event]	[Service]	[Service]	[Impact]	[Impact]	[Semantics]

2. Delivery semantics

Semantics	Meaning	How it arises	Use when
At-most-once	Each message delivered zero or one time	Acknowledge before processing, no retries	Loss is cheap (telemetry, some analytics)
At-least-once	Each message delivered one or more times	Acknowledge after processing, retry on doubt	Default for business events
Effectively-once (exactly-once processing)	The effect of each message is applied once, even if delivered more than once	At-least-once delivery plus an idempotent consumer, or a transactional read-process-write inside one system	Money, inventory, anything counted

The Apache Kafka documentation (4.4) explains why: when a producer gets no reply it cannot know whether the write was committed, so resending gives at-least-once behaviour. The idempotent producer removes duplicates caused by producer retries within the log; transactions let a read-process-write application commit its output and its input offsets together, and consumers in read_committed isolation see only committed data. The same documentation notes that writing to an external system needs that system's cooperation to coordinate output with the consumer position.

3. Publishing: transactional outbox

Decision	Options	Example choice
Relay type	Polling publisher, or transaction log tailing (change data capture)	Log tailing via the CDC pipeline (`system-design/design-cdc-pipeline`)
Event ID	UUID or database sequence, assigned in the transaction	UUIDv7 assigned by the service
Ordering	Per aggregate: partition key = aggregate ID	Partition key `book_id`
Outbox cleanup	Delete or archive published rows after a retention period	Delete after 3 days
Volume	Average and peak events per second	20,000,000 per day, about 231 per second on average

4. Idempotent consumers

Technique	How it works	Fits
Natural idempotency	The operation gives the same result when repeated (set status, upsert by key)	State updates
Dedup table	Record processed event IDs in the same transaction as the effect; skip known IDs	Additive effects (send, count, charge)
Version check	Apply only if the event's version is newer than the stored one	Out-of-order updates to one entity
Idempotency key at the target	Pass a key to the downstream API so it rejects repeats	Payments and external APIs

Dedup table sizing, Example: 20,000,000 events per day kept for 7 days is 140,000,000 rows; at about 64 bytes per row including index overhead that is 8.96 GB. The retention must exceed the longest redelivery window (broker retention plus replay periods), or a late duplicate is applied again.

5. Ordering, retries and dead letters

Rule	Example decision
Ordering scope	Per `book_id` (one partition per key)
Retry policy	Exponential backoff 1, 2, 4, 8, 16, 32 s then 60 s cap; first attempt plus 8 retries, 183 s of waiting in total
Poison message	After the last retry, move to a dead-letter topic with error, attempt count and original offset
Ordering versus retry	Retrying event moves to a retry topic; later events for the same key wait behind it (key is parked)
Dead-letter handling	Alert on any message; owning team triages within 1 business day; replay tool re-injects after fix
Schema evolution	Consumers ignore unknown fields; breaking changes use a new event type or version

Backoff schedule, Example:

Retry	1	2	3	4	5	6	7	8
Wait before retry (s)	1	2	4	8	16	32	60	60
Cumulative wait (s)	1	3	7	15	31	63	123	183

Add random jitter to each wait in production so that many consumers do not retry in step; see the Resilience and Load Shedding Design template (system-design/resilience-load-shedding-design).

6. Exactly-once: limits

Boundary	Effectively once?	Why
Broker topic to broker topic with transactional read-process-write	Yes, within the broker	Output and input offsets committed together
Broker to consumer database, offset stored in the same database transaction	Yes	Effect and position committed atomically
Broker to consumer database, offset stored in the broker	No, at-least-once	Crash between database commit and offset commit replays
Consumer sends email or push notification	No, at-least-once	External system cannot join the transaction; dedup by event ID before sending narrows but does not close the gap
Consumer calls a payment API with an idempotency key	Yes, if the API honours the key for the retry window	Target rejects repeats

7. Sagas for multi-service transactions

Example: gift purchase saga (orchestrated).

Step	Service	Action	Compensation	Can it be undone?
1	Orders	Create gift order (pending)	Cancel order	Yes
2	Payments	Authorise card	Void authorisation	Yes
3	Inventory	Reserve e-book licence	Release licence	Yes
4	Payments	Capture payment	Refund	Yes, with fees
5	Notifications	Email gift code to recipient	None (cannot unsend)	No, so it runs last

For long-running orchestration with timers and durable state, see the Workflow Orchestration Engine System Design template (system-design/design-workflow-orchestration-engine).

8. Monitoring

Signal	Example alert
Outbox backlog (unpublished rows, oldest age)	Oldest unpublished older than 60 s
Consumer lag per group and partition	Lag above 5 minutes
Dead-letter count	Any new message
Duplicate rate (dedup table hits per second)	Above 1% of deliveries for 15 minutes
End-to-end reconciliation	Daily count of `review.posted` events vs new reviews differs by more than 0
Saga stuck in a non-final state	Older than 30 minutes

9. Decisions and open items

ID	Decision or open item	Reason	Owner	Status
ED-01	Example: All business events via transactional outbox	No lost or phantom events on crash	@platform-events-lead	Decided
ED-02	Example: Consumers keep a 7-day dedup table for additive effects	Broker retention is 3 days, replays within 7	@notifications-lead	Decided
ED-03	Example: Gift purchase as orchestrated saga, email last	Email cannot be compensated	@orders-lead	Decided
ED-04	Example: Decide whether analytics may drop events under load	At-most-once acceptable for page views	@analytics-lead	Open
[ED-NN]	[Item]	[Reason]	[@owner]	[Status]

Glossary and references

Term	Meaning
Outbox	Table written in the same transaction as the business change, holding events to publish
Relay	Process that reads the outbox and publishes to the broker
Idempotent	Applying an operation more than once has the same effect as once
Dead-letter topic	Destination for messages that failed all retries
Saga	Sequence of local transactions with compensating actions instead of one distributed transaction
Compensation	Action that semantically undoes an earlier step

References:

Richardson C., "Pattern: Transactional outbox", microservices.io, https://microservices.io/patterns/data/transactional-outbox.html (checked 2026-10-30).
Garcia-Molina H., Salem K., "Sagas", Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data, pages 249 to 259, https://doi.org/10.1145/38713.38742.
Helland P., "Life Beyond Distributed Transactions", ACM Queue 14(5), 2016, pages 69 to 98, https://doi.org/10.1145/3012426.3025012.
Apache Kafka 4.4 documentation, Design, delivery semantics, https://kafka.apache.org/44/design/design/, and producer configuration enable.idempotence, https://kafka.apache.org/44/configuration/producer-configs/ (checked 2026-10-30).
Related templates: Event Sourcing and CQRS Design (architecture/event-sourcing-cqrs), Event-Driven Architecture Design (architecture/event-driven-architecture-design), Message Queue System Design (system-design/design-message-queue), Webhook Delivery System Design (system-design/design-webhook-delivery), Payment Ledger System Design (system-design/design-payment-ledger).

Standard and licence note

Based on the publicly available papers, the Apache Kafka documentation and the pattern description cited above; no standard governs event delivery between services. The text, tables, numbers and examples are original to this collection, and Quillmere is an invented organisation. Broker features differ and change between versions: check the current documentation of the broker and client libraries you use. Template licence: MIT.