# System Design: [Message queue or event log name]

> Reference design for a distributed message queue built as a partitioned,
> replicated, append-only log. It covers topics and partitions, log
> segments and retention, producers (acknowledgements and idempotence),
> consumer groups and offset management, delivery semantics and their real
> limits, ordering guarantees, replication with in-sync replicas and leader
> election, backpressure, retries and dead-letter topics, schema
> management and throughput estimation. Concepts and default values follow
> the Apache Kafka 4.3 documentation and the original Kafka paper (Kreps,
> Narkhede and Rao, NetDB 2011); the design applies to any comparable log.
> It follows the structure of the System Design Document template and is
> pre-filled with a worked design to adapt. Values marked *Example* are
> assumptions. Replace every `[bracketed]` value and every row marked
> *Example*.

<!--
Guidance: How to use this template
- Decide first which delivery guarantee each consumer really needs, end to
  end, including its database or external calls. "Exactly once" inside the
  log does not make an email or a payment happen exactly once (7.6).
- Choose partition keys before partition counts: the key decides ordering,
  hot partitions and whether you can ever change the count (7.7).
- Size from section 3 with your own message sizes and retention; storage is
  usually the binding constraint, not network.
- Broker defaults are not production settings. Review the table in 7.4 and
  7.8 against your version's documentation.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-MQ-001] |
| System / feature | [Event streaming platform] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Log technology and version | [Apache Kafka 4.x / managed service / other] |
| Related | [Event API specification], [Capacity estimation worksheet CAP-EST-nnn], [ADR: delivery semantics] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| Data governance (schemas, retention) | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Raised `min.insync.replicas` to 2 for all business topics |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added retry topics and dead-letter topics per consumer group |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Services integrate through synchronous calls and a shared
database. An outage of the order service blocks checkout, analytics reads
production replicas, and three teams run their own queues with different
guarantees. Peak traffic during sales reaches 1 million events per second
and nobody can replay events after a consumer bug.]

### 1.2 Proposed solution

[*Example:* A shared event log platform. Producers write keyed events to
topics split into partitions; each partition is an append-only log of
segments replicated to three brokers in three zones. Producers use
`acks=all` with idempotence; topics require two in-sync replicas. Consumers
form groups, each partition is read by one member of a group, and offsets
are committed after processing. Events are retained for 7 days for replay;
compacted topics hold the latest state per key. Schemas are registered and
checked for compatibility before a producer can publish. Failed messages go
through delayed retry topics and finally a dead-letter topic owned by the
consuming team.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Default delivery guarantee for consumers? | At-least-once with idempotent consumers; transactions only for log-to-log pipelines | 7.6 |
| Q-02 | *Example:* Durability settings? | Replication factor 3, `min.insync.replicas=2`, `acks=all`, unclean election off | 7.8 |
| Q-03 | *Example:* Retention for business topics? | 7 days delete plus compacted state topics | 7.2 |
| Q-04 | *Example:* Run ourselves or use a managed service? | [Decision] | 10 |
| [Q-nn] | [Question] | [Option] | [Section] |

## 2. Requirements

<!-- Guidance: Functional requirements describe behaviour; non-functional requirements describe qualities with numbers. Mark what is out of scope so nobody designs for it. -->

### 2.1 Functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Publish keyed messages to named topics with headers | Must |
| FR-02 | *Example:* Ordered delivery per key | Must |
| FR-03 | *Example:* Several independent consumer groups per topic, each with its own position | Must |
| FR-04 | *Example:* Replay from an offset or timestamp within retention | Must |
| FR-05 | *Example:* Compacted topics keeping the latest value per key | Should |
| FR-06 | *Example:* Retry with delay and dead-letter handling per consumer group | Must |
| FR-07 | *Example:* Schema registration and compatibility checks | Must |
| FR-08 | *Example:* Work-queue consumption with per-message acknowledgement | Could |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Throughput | *Example:* 1 million messages/s (1 GB/s) at peak | Load test |
| NFR-02 | Publish latency | *Example:* p99 at or below 30 ms with `acks=all` | Producer metrics |
| NFR-03 | End-to-end latency | *Example:* p99 at or below 200 ms from publish to consumer poll | Timestamp headers |
| NFR-04 | Durability | *Example:* No acknowledged message lost with one broker or one zone lost | Fault injection with verification |
| NFR-05 | Availability | *Example:* Writes available with one zone lost | Zone failure game day |
| NFR-06 | Recovery | *Example:* A consumer group recovers a 30-minute backlog within 15 min | Lag drain test |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Out of scope

- [*Example:* cross-region active-active replication (later phase, mirrored topics)]
- [*Example:* stream processing frameworks; teams choose their own]
- [*Example:* messages larger than 1 MB (use object storage and send a reference)]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. The method is in the Capacity estimation worksheet (`architecture/back-of-envelope-estimation`). -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| Peak messages per second (all topics) | $M_{p}$ | *Example:* 1 000 000 | Sale-day traffic x growth |
| Average messages per second | $M_{a}$ | *Example:* 400 000 | Daily curve |
| Average message size on the wire, compressed, with overhead | $m$ | *Example:* 1 KB | Producer metrics |
| Replication factor | $f$ | *Example:* 3 | 7.8 |
| Consumer groups reading all data | $g$ | *Example:* 3 | Consumer inventory |
| Retention | $t$ | *Example:* 7 days = 604 800 s | 7.2 |
| Usable disk per broker at 70% fill (24 TB raw) | $d$ | *Example:* 16.8 TB | Hardware standard |
| Throughput per partition, producer side | $p_{w}$ | *Example:* 10 MB/s | Benchmark |
| Throughput per partition, slowest consumer | $p_{c}$ | *Example:* 5 MB/s (5 000 messages/s) | Consumer benchmark |

### 3.2 Throughput and network

$$
T_{peak} = M_{p} \times m = 10^{6} \times 1 \ \text{KB} = 1 \ \text{GB/s}, \qquad T_{avg} = 4 \times 10^{5} \times 1 \ \text{KB} = 400 \ \text{MB/s}
$$

At peak, brokers receive producer traffic plus follower replication, and
send consumer traffic plus replication:

$$
B_{in} = T_{peak} \times f = 3 \ \text{GB/s}, \qquad
B_{out} = T_{peak} \times g + T_{peak} \times (f - 1) = 3 + 2 = 5 \ \text{GB/s}
$$

### 3.3 Storage and broker count

$$
S = T_{avg} \times t \times f = 400 \ \text{MB/s} \times 604\,800 \ \text{s} \times 3 \approx 725.8 \ \text{TB}
$$

$$
n = \left\lceil \frac{725.8}{16.8} \right\rceil = \lceil 43.2 \rceil = 44 \Rightarrow 45 \ \text{brokers (15 per zone)}
$$

Per broker at peak this is about $3\,000 / 45 \approx 67$ MB/s in and
$5\,000 / 45 \approx 111$ MB/s out, well under a 25 Gbit/s (about 3.1 GB/s)
network interface. Storage, not network, sets the broker count; tiered
storage (7.2) would change that.

### 3.4 Partitions

The partition count must let both producers and the slowest consumer group
keep up, with headroom $h$ for growth:

$$
P = \max\left(\frac{T_{peak}}{p_{w}}, \frac{T_{peak}}{p_{c}}\right) \times h = \max(100, 200) \times 1.5 = 300
$$

With replication factor 3 that is 900 partition replicas, 20 per broker.
Apply the same formula per topic with that topic's own peak.

### 3.5 Backlog recovery

A consumer group with $P$ members at $5\,000$ messages/s each processes
$1.5 \times 10^{6}$ messages/s. After a 30-minute outage at average load the
backlog is $L = 4 \times 10^{5} \times 1\,800 = 7.2 \times 10^{8}$ messages, and
with new messages still arriving at $\lambda = 4 \times 10^{5}$/s:

$$
t_{drain} = \frac{L}{C - \lambda} = \frac{7.2 \times 10^{8}}{1.5 \times 10^{6} - 4 \times 10^{5}} \approx 655 \ \text{s} \approx 11 \ \text{min}
$$

This meets NFR-06. Catch-up reads come from disk rather than page cache, so
schedule the drain test on production-sized brokers.

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Ingress | 1 GB/s peak, 400 MB/s average | Batching and compression at producers |
| Broker network | 3 GB/s in, 5 GB/s out cluster-wide | About 67 MB/s in and 111 MB/s out per broker |
| Storage | about 726 TB for 7 days with 3 replicas | 45 brokers, or fewer with tiered storage |
| Partitions | 300 across business topics | 20 replicas per broker |
| Backlog drain | about 11 min for 30 min of lag | Consumers sized at 1.5 times peak partition need |

## 4. API design

<!-- Guidance: Define the contract first: operations, configuration that changes semantics, errors and versioning. Publish the topic contract (schema, key, ordering, retention) in the event API specification. -->

### 4.1 Operations

| Operation | Caller | Semantics |
|---|---|---|
| `produce(topic, key, value, headers)` | Producer | Appends to the partition chosen by key hash; returns partition and offset once acknowledged per `acks` |
| `fetch(topic, partition, offset, max_bytes)` | Consumer | Pull-based read from an offset; long-polls until data or timeout |
| `commit(group, partition, offset)` | Consumer | Records the next offset to read for the group |
| `begin`, `send_offsets`, `commit_txn`, `abort_txn` | Transactional producer | Atomic writes across partitions plus consumer offsets |
| `create_topic(name, partitions, config)` | Platform API | Through a reviewed request, not ad hoc |

### 4.2 Producer and consumer configuration

The defaults below are from the Apache Kafka 4.3 configuration reference;
check the documentation for your version.

| Setting | Kafka 4.3 default | Platform setting | Why |
|---|---|---|---|
| `acks` | `all` | `all` | Acknowledge only when all in-sync replicas have the record |
| `enable.idempotence` | `true` (if no conflicting settings) | `true` | Broker drops duplicate retries using producer ID and sequence numbers |
| `max.in.flight.requests.per.connection` | 5 | 5 | Highest value that keeps ordering with idempotence |
| `linger.ms` | 5 | 10 *(Example)* | Larger batches, better compression |
| `batch.size` | 16 384 bytes | 131 072 *(Example)* | Fewer requests at high throughput |
| `compression.type` | `none` | `zstd` or `lz4` *(Example)* | Cuts disk and network |
| Consumer `enable.auto.commit` | `true` | `false` | Commit after processing, not on a timer |
| Consumer `isolation.level` | `read_uncommitted` | `read_committed` for transactional topics | Hide aborted transactional records |

### 4.3 Message envelope

```json
{"key": "order-7f3a91", "headers": {"event_type": "OrderPlaced", "schema_id": "1042", "event_id": "0b6e6c8e-6f0e-4b0a-9e1d-2f3c4d5e6f70", "produced_at": "YYYY-MM-DDTHH:MM:SSZ", "trace_parent": "00-[trace-id]-[span-id]-01"}, "value": {"order_id": "7f3a91", "customer_id": "c-2231", "total_minor": 4599, "currency": "EUR"}}
```

`event_id` is the consumer's idempotency key; the value is serialised with
the registered schema (7.11).

## 5. Data model

<!-- Guidance: Show the logical objects (topics, partitions, segments, groups, offsets) and the per-topic contract that teams must fill in. -->

### 5.1 Entities

A topic has partitions; each partition replica is a sequence of segments on
one broker; groups track offsets per partition.

```mermaid
erDiagram
  TOPIC {
    string name PK
    int partitions
    int replication_factor
    string cleanup_policy
    bigint retention_ms
    string owner_team
  }
  PARTITION {
    string topic FK
    int partition_id PK
    int leader_broker
    string isr
    bigint high_watermark
  }
  SEGMENT {
    string topic FK
    int partition_id FK
    bigint base_offset PK
    bigint size_bytes
    bool active
  }
  CONSUMER_GROUP {
    string group_id PK
    string state
    int generation
  }
  COMMITTED_OFFSET {
    string group_id FK
    string topic FK
    int partition_id FK
    bigint offset
  }
  TOPIC ||--o{ PARTITION : "split into"
  PARTITION ||--o{ SEGMENT : "stored as"
  CONSUMER_GROUP ||--o{ COMMITTED_OFFSET : tracks
  PARTITION ||--o{ COMMITTED_OFFSET : "position in"
```

### 5.2 Topic contract

| Topic | Key | Partitions | Cleanup | Retention | Schema | Owner | Consumers |
|---|---|---|---|---|---|---|---|
| `orders.events.v1` *(Example)* | `order_id` | 60 | delete | 7 days | `OrderEvent` (Avro) | Orders | Payments, fulfilment, analytics |
| `customers.state.v1` *(Example)* | `customer_id` | 30 | compact | Latest per key | `Customer` (Protobuf) | Customers | Search, CRM sync |
| `payments.retry.5m` *(Example)* | Original key | 12 | delete | 1 day | Same as source | Payments | Payments consumer |
| `payments.dlq` *(Example)* | Original key | 6 | delete | 30 days | Same as source plus error headers | Payments | Payments on-call |
| [Topic] | [Key] | [n] | [Policy] | [Retention] | [Schema] | [Team] | [Groups] |

Topic names carry a major version; a breaking schema change creates a new
topic and a migration window, never an in-place break.

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

```mermaid
flowchart LR
  prod([Producers]) --> sr[(Schema registry)]
  prod --> b1
  subgraph cluster["Broker cluster, 45 brokers in 3 zones"]
    b1[Broker: partition leader] --> f1[Follower, zone b]
    b1 --> f2[Follower, zone c]
    ctrl[KRaft controller quorum] -.-> b1
  end
  b1 --> cg1[Consumer group: payments]
  b1 --> cg2[Consumer group: fulfilment]
  b1 --> cg3[Consumer group: analytics]
  cg1 -->|failure| retry[[Retry topics]]
  retry --> cg1
  retry -->|attempts exhausted| dlq[[Dead-letter topic]]
  cg1 --> sr
  b1 -.-> tier[(Tiered object storage, optional)]
  ops[Platform API, quotas, ACLs] --> ctrl
```

| Component | Responsibility | Scales by |
|---|---|---|
| Brokers | Store partition replicas, serve produce and fetch | Brokers and disks |
| Controller quorum | Metadata, leader election, ISR changes (KRaft; Kafka 4.0 and later have no ZooKeeper mode) | Three or five controllers |
| Schema registry | Store schemas, enforce compatibility | Small replicated service |
| Consumer groups | Process partitions; one member per partition per group | Members up to partition count |
| Retry and dead-letter topics | Delay and park failed messages | Per consumer group |
| Platform API | Topic creation, ACLs, quotas, ownership | Single service |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

### 7.1 Topics and partitions

A topic is a named stream; a partition is an ordered, append-only sequence
in which each record gets a monotonically increasing offset. Partitions are
the unit of parallelism, ordering, replication and placement. The producer
chooses the partition by hashing the key; records without a key are spread
across partitions in batches.

### 7.2 Log segments and retention

Each partition replica is a directory of segment files, each with an index
by offset and an index by timestamp. Only the newest (active) segment is
written; older segments are immutable. Retention deletes whole closed
segments, which is why it is cheap and why data can live slightly longer
than the configured retention.

- **Time and size retention:** Kafka 4.3 defaults to 168 hours
  (`log.retention.hours`) and rolls segments at 1 GiB
  (`log.segment.bytes`). At the example average of about 1.33 MB/s per
  partition (400 MB/s over 300 partitions), a segment fills in about
  $1\,073\,741\,824 / (1.33 \times 10^{6}) \approx 805$ s, so retention
  granularity is fine.
- **Compaction:** with `cleanup.policy=compact` the log keeps at least the
  latest value per key; a null value is a tombstone that is removed after
  `delete.retention.ms` (default 24 hours). Use it for state topics, not for
  event histories.
- **Tiered storage:** recent Kafka releases can offload closed segments to
  object storage, keeping only recent data on broker disks. It cuts the
  storage-driven broker count in section 3.3; check the feature status and
  limitations for your version before relying on it.

### 7.3 Produce path

One batch from an idempotent producer with `acks=all`, including the retry case:

```mermaid
sequenceDiagram
  participant P as Producer
  participant L as Leader (zone a)
  participant F1 as Follower (zone b)
  participant F2 as Follower (zone c)
  P->>L: ProduceRequest batch (producer id, seq 41..60, acks=all)
  L->>L: check sequence, append to active segment
  F1->>L: fetch from offset
  F2->>L: fetch from offset
  L-->>F1: records
  L-->>F2: records
  F1->>L: next fetch confirms replication
  F2->>L: next fetch confirms replication
  L->>L: advance high watermark
  L-->>P: ack (partition 17, base offset 9 120 334)
  Note over P,L: on timeout the producer retries the same sequence; the leader drops the duplicate
```

### 7.4 Producers: acknowledgements and idempotence

| `acks` | Acknowledged when | Loss risk | Latency |
|---|---|---|---|
| `0` | Sent, no broker response | Any failure loses data | Lowest |
| `1` | Leader has written the record | Loss if the leader fails before followers copy it | Low |
| `all` | All current in-sync replicas have the record | None while at least `min.insync.replicas` copies exist | Highest |

Idempotence (available since Kafka 0.11, on by default in current
releases) gives each producer an ID and numbers each batch per partition, so
the leader discards duplicates from retries. It removes duplicates caused by
the producer's own retries within a producer session; it does not
deduplicate an application that sends the same business event twice, which
is why events also carry `event_id`.

### 7.5 Consumer groups and offset management

Members of a group divide the partitions of the subscribed topics; each
partition has exactly one member at a time. Committed offsets are stored as
records in an internal compacted topic (`__consumer_offsets` in Kafka), so
they are replicated like data. Kafka 4.0 made the new server-driven
rebalance protocol (KIP-848) generally available, which reduces
stop-the-world rebalances; Java consumers opt in with
`group.protocol=consumer`, and other clients may lag behind, so check the
client support for your language.

Commit rules for the platform:

1. Disable auto commit; commit after the processing side effects are
   durable.
2. Commit the offset of the next record to read, per partition.
3. On rebalance, finish or abandon in-flight work and commit before
   releasing partitions.
4. Consumers must tolerate seeing a record again after a crash (7.6).

For work-queue use where many consumers share a partition with
per-message acknowledgement, Kafka share groups (KIP-932) were declared
production-ready in Kafka 4.2; confirm client support before choosing them
over classic consumer groups.

### 7.6 Delivery semantics and their real limits

| Semantics | Mechanism | What it guarantees | What it does not |
|---|---|---|---|
| At-most-once | Commit offset before processing | No duplicates | Messages lost if the consumer crashes mid-processing |
| At-least-once | Commit after processing | No loss within retention | Duplicates after crashes, rebalances or retries |
| Exactly-once within the log | Idempotent producer plus transactions; consumers in `read_committed`; offsets committed in the same transaction | A read-process-write pipeline from topic to topic produces each output once, with its input offset | Any side effect outside the log: database writes, HTTP calls, emails |
| Effectively-once end to end | At-least-once plus idempotent processing (dedupe on `event_id`, upserts, conditional writes) or a transactional outbox | Each business effect applied once | Requires discipline in every consumer |

The platform default is at-least-once with idempotent consumers. Transactions
are used for stream processing jobs that read and write topics (for example
Kafka Streams with exactly-once processing enabled). Transactional producers
need a stable `transactional.id` so that a restarted instance fences the old
one; a transaction timeout aborts work from a producer that disappeared.

### 7.7 Ordering guarantees

- Order is guaranteed only within a partition. Records with the same key
  go to the same partition while the partition count stays the same.
- With idempotence and up to 5 in-flight requests, retries do not reorder
  records within a partition.
- Increasing the partition count changes the key-to-partition mapping for
  new records, so per-key order breaks across the change. Over-provision
  partitions at creation (3.4) or create a new topic and migrate.
- A consumer that processes records of one partition in parallel threads
  must keep per-key order itself (for example, one queue per key hash).
- Retry topics break order for the failed key; topics that require strict
  per-key order pause the partition instead of forwarding to retry.

### 7.8 Replication, in-sync replicas and leader election

Each partition has one leader and followers that fetch from it. The leader
tracks the in-sync replica set (ISR): replicas that keep an active session
with the controller and are not too far behind (Kafka 4.3 default for
`replica.lag.time.max.ms` is 30 s). A record is committed when all ISR
members have it, and consumers only read up to the high watermark.

| Setting | Kafka 4.3 default | Platform setting | Effect |
|---|---|---|---|
| `default.replication.factor` | 1 | 3 | One replica per zone |
| `min.insync.replicas` | 1 | 2 | With `acks=all`, writes fail if fewer than 2 replicas are in sync |
| `unclean.leader.election.enable` | `false` | `false` | Never elect an out-of-sync replica, which could lose committed data |
| `offsets.topic.replication.factor` | 3 | 3 | Offsets as durable as data |

With replication factor 3 and `min.insync.replicas=2`, one broker or one
zone can be lost without losing acknowledged data or write availability; a
second simultaneous loss in the same partition stops writes (producers get
a "not enough replicas" error) rather than risking loss. The controller
quorum elects a new leader from the ISR when a leader fails.

### 7.9 Backpressure

The log decouples rates, but every buffer is finite:

| Point | Mechanism | Platform setting |
|---|---|---|
| Producer | Bounded send buffer; `send` blocks then fails when full | Fail fast after a short block; shed or spill non-critical events |
| Broker | Produce and fetch byte-rate quotas per client or user; request throttling | Quotas per team from the capacity plan |
| Consumer | Pull model; bounded records per poll; pause and resume partitions | Pause when downstream is saturated; alert on lag |
| Downstream | Consumer rate limits to protect databases and third-party APIs | Token bucket per dependency |

Consumer lag (latest offset minus committed offset) is the main signal. For
timeouts, retry budgets and load-shedding policies across services, align
with the Resilience and load shedding design
(`architecture/resilience-load-shedding-design`).

### 7.10 Retries and dead-letter topics

Most log technologies have no built-in dead-letter topic for plain
consumers, so the platform provides a library:

1. Classify the error: transient (timeout, 503) or permanent (validation,
   unknown schema).
2. Transient: retry in-process up to 3 times with jittered backoff, then
   publish to a delay topic (`retry.5m`, then `retry.1h`) with headers for
   the original topic, partition, offset, attempt and error.
3. Permanent, or attempts exhausted: publish to the group's dead-letter
   topic and commit the original offset.
4. The owning team triages the dead-letter topic with a replay tool that
   republishes selected records to the original topic or a retry topic.

Dead-letter topics count as production data: same access controls,
retention of 30 days and an alert when the rate exceeds a threshold.

### 7.11 Schema registry and compatibility

Producers register Avro, Protobuf or JSON Schema definitions and send the
schema ID in each message; consumers fetch schemas by ID and cache them.
The registry rejects incompatible changes according to the topic's mode:

| Mode | Allowed change | Upgrade order |
|---|---|---|
| Backward | New schema can read data written with the previous one (add optional fields, remove fields) | Consumers first |
| Forward | Previous schema can read data written with the new one | Producers first |
| Full | Both directions | Either |
| None | Anything | Not allowed for shared topics |

The platform default is backward compatibility with transitive checks for
shared topics. Document each topic's contract in the event API specification
(`architecture/event-api-specification`).

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Throughput | Add brokers and partitions; producer batching and compression | *Example:* per-broker network above 50% of NIC |
| Storage | 7-day retention; tiered storage for longer replay | Disk fill above 70% |
| Partition count | Over-provision at creation; cap per broker | *Example:* several thousand replicas per broker |
| Hot partition | Better key choice, key salting for non-ordered data | One key above a partition's throughput |
| Consumer parallelism | Members up to partition count; per-key parallel processing inside members | Partition count |

Latency budget for publish with `acks=all` (target p99 30 ms):

| Step | Budget |
|---|---|
| Producer linger and batching | 10 ms |
| Network to leader | 1 ms |
| Append to page cache | 1 ms |
| Follower fetch round trip across zones (two followers in parallel) | 5 ms |
| Acknowledgement to producer | 1 ms |
| Headroom (garbage collection, queueing, disk flush) | 12 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Failure | Impact | Detection | Mitigation |
|---|---|---|---|
| Broker down | Leaders move; ISR shrinks | Under-replicated partitions | Leader election from ISR; replace broker |
| Zone down | One third of brokers gone; ISR at 2 | Zone health, under-replicated partitions | `min.insync.replicas=2` keeps writes up; rack-aware placement |
| Slow follower | Leaves ISR; durability margin shrinks | ISR shrink rate | Investigate disk or network; throttle reassignments |
| Consumer bug writes bad data | Downstream corruption | Data quality checks | Reset group offsets to a timestamp and replay |
| Poison message | Consumer stuck on one offset | Lag growing on one partition | Retry and dead-letter path (7.10) |
| Controller quorum loss | No leader elections or metadata changes | Controller health | Five controllers across three zones; tested recovery runbook |
| [Failure] | [Impact] | [Signal] | [Mitigation] |

### 9.2 Security

| Threat | Control |
|---|---|
| Unauthorised produce or consume | mTLS or SASL authentication; ACLs per topic and group, owned by teams |
| Sensitive data in events | Data classification per topic; field-level encryption for regulated fields; no secrets in headers |
| Data retained too long | Retention per topic reviewed by data governance; compaction tombstones for erasure |
| Schema poisoning | Registry writes limited to producer service accounts; compatibility enforced |
| Noisy neighbour | Quotas per client and team |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Under-replicated partitions | Count per cluster | Any for more than 5 min |
| Offline partitions | Count | Any |
| Consumer lag | Messages and seconds behind, per group | Lag in seconds above the group's SLO |
| Produce errors | Error rate by type | Above 0.1% for 5 min |
| Dead-letter rate | Messages per minute per group | Above the group threshold |
| Disk usage | Per broker | Above 70% |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Brokers | [USD per broker-month] | *Example:* 45 | [USD] |
| Controllers | [USD per node-month] | *Example:* 5 | [USD] |
| Cross-zone replication traffic | [USD per GB] | $2 \times 400$ MB/s average | [USD] |
| Schema registry and platform API | [USD per month] | 1 deployment | [USD] |
| Tiered object storage (if used) | [USD per TB-month] | [TB] | [USD] |
| **Total** | | | **[USD]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Model | Broker-centric queue (per-message ack, deleted on consume); partitioned log | Partitioned log | Replay, many independent consumers, high throughput | Per-message routing features; work-queue semantics need share groups |
| Delivery default | At-most-once; at-least-once; exactly-once everywhere | At-least-once plus idempotent consumers | Works with external side effects | Every consumer must deduplicate |
| Durability | `acks=1`; `acks=all` with `min.insync.replicas=2` | `acks=all`, ISR 2 | No loss on one failure | Higher publish latency |
| Retention | Short; long on brokers; tiered | 7 days on brokers, tiered later | Replay window for incidents | Disk cost drives broker count |
| Operation | Self-managed; managed service | [Choice] | [Reason] | [Cost] |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Cluster, platform API, schema registry, one pilot topic | Fault injection suite passes; no acknowledged loss | YYYY-MM-DD |
| M2 | Retry and dead-letter library, consumer lag dashboards | Poison-message test recovers without manual offset edits | YYYY-MM-DD |
| M3 | Orders and payments topics in production | NFR-02 and NFR-03 met for 14 days | YYYY-MM-DD |
| M4 | Teams' legacy queues migrated | Legacy queues decommissioned | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Producers dual-publish to the legacy queue and the new topic;
consumers move one group at a time with a flag. Rollback flips the consumer
flag back; dual publishing continues until 30 days after the last consumer
moved.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Consumers assume exactly-once and double-apply effects | High | High | Idempotency checklist in consumer reviews; dedupe library | [@name] |
| R-02 | *Example:* Partition count too low, re-keying needed | Medium | High | Over-provision per 3.4; capacity review each quarter | [@name] |
| R-03 | *Example:* Schema change breaks a consumer | Medium | Medium | Registry compatibility checks in CI | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Which topics need cross-region mirroring, and with what lag? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Adopt share groups for the job-processing workload? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| Partition | Ordered, append-only log that is the unit of parallelism and replication |
| Offset | Position of a record within a partition |
| Segment | One file of a partition's log; retention deletes whole segments |
| ISR | In-sync replicas: replicas caught up closely enough to the leader |
| High watermark | Highest offset replicated to all ISR members; consumers read up to it |
| Consumer group | Set of consumers that share the partitions of a subscription |
| Consumer lag | Difference between the latest offset and the group's committed offset |
| Dead-letter topic | Topic that holds messages a consumer could not process |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| Apache Kafka 4.3 documentation: design, producer, consumer and broker configuration | Concepts and default values in 4.2, 7.2, 7.4, 7.6 and 7.8 |
| J. Kreps, N. Narkhede, J. Rao, "Kafka: a Distributed Messaging System for Log Processing", NetDB 2011 | Log-based design, pull consumers, partitioned topics |
| Apache Kafka 4.0 and 4.2 release announcements | KRaft-only operation, KIP-848 rebalance protocol, share groups (KIP-932) |
| Event API specification (`architecture/event-api-specification`) | Topic contracts and schemas |
| Capacity estimation worksheet (`architecture/back-of-envelope-estimation`) | Method for section 3 |
| Capacity plan (`ops/capacity-plan`) | Ongoing broker and partition capacity |

## Standard and licence note

Based on the Apache Kafka 4.3 documentation and the Kafka paper by Kreps,
Narkhede and Rao (2011), used as concept sources, and on common industry
practice for event streaming platforms. Default values are cited as
published for Kafka 4.3 in October 2026; check the documentation for the
version you run, because defaults change between releases. This is a
reference design to adapt, with illustrative numbers. Original text written
for this collection; no documentation text is reproduced. Apache Kafka is a
trademark of the Apache Software Foundation. Template licence: MIT.