Distributed Message Queue System Design Template (Kafka-Style Log)

Partitions, log segments, retention, acks and idempotence, consumer groups, offsets, delivery semantics, ordering, ISR replication, backpressure, DLQ, schemas

Conception systèmeCompletApache Kafka 4.3 documentation; Kafka paper (NetDB 2011)design message queue mathMis à jour 22 oct. 2026

Ouvrir dans l'éditeurTélécharger .md

Mode d'emploi: Use it when designing an event streaming platform or a shared queue. Choose partition keys and the end-to-end delivery guarantee per consumer first, then size partitions and brokers from section 3. The main pitfall is believing log-level exactly-once covers database writes and external calls; make consumers idempotent.

Aperçu

System Design: [Message queue or event log name]

Reference design for a distributed message queue built as a partitioned, replicated, append-only log. It covers topics and partitions, log segments and retention, producers (acknowledgements and idempotence), consumer groups and offset management, delivery semantics and their real limits, ordering guarantees, replication with in-sync replicas and leader election, backpressure, retries and dead-letter topics, schema management and throughput estimation. Concepts and default values follow the Apache Kafka 4.3 documentation and the original Kafka paper (Kreps, Narkhede and Rao, NetDB 2011); the design applies to any comparable log. It follows the structure of the System Design Document template and is pre-filled with a worked design to adapt. Values marked Example are assumptions. Replace every [bracketed] value and every row marked Example.

Document control

Field	Value
Document ID	[SDD-MQ-001]
System / feature	[Event streaming platform]
Version	[0.1]
Status	[Draft / In review / Approved / Implemented / Superseded]
Author(s)	[@author]
Reviewers	[@reviewer-1], [@reviewer-2]
Last updated	YYYY-MM-DD
Classification	[Internal]
Log technology and version	[Apache Kafka 4.x / managed service / other]
Related	[Event API specification], [Capacity estimation worksheet CAP-EST-nnn], [ADR: delivery semantics]

Approvals

Role	Name	Decision	Date
Tech lead	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Staff / principal engineer	[@name]	[Decision]	YYYY-MM-DD
SRE / operations	[@name]	[Decision]	YYYY-MM-DD
Security	[@name]	[Decision]	YYYY-MM-DD
Data governance (schemas, retention)	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft for design review
0.2	YYYY-MM-DD	[@name]	Example: Raised `min.insync.replicas` to 2 for all business topics
0.3	YYYY-MM-DD	[@name]	Example: Added retry topics and dead-letter topics per consumer group
1.0	YYYY-MM-DD	[@name]	Example: Approved

Overview
Requirements
Capacity estimation
API design
Data model
High-level design
Detailed design
Scalability and performance
Reliability, security and operations
Trade-offs and alternatives
Delivery plan and risks
Appendices

1. Overview

1.1 Problem statement

[Example: Services integrate through synchronous calls and a shared database. An outage of the order service blocks checkout, analytics reads production replicas, and three teams run their own queues with different guarantees. Peak traffic during sales reaches 1 million events per second and nobody can replay events after a consumer bug.]

1.2 Proposed solution

[Example: A shared event log platform. Producers write keyed events to topics split into partitions; each partition is an append-only log of segments replicated to three brokers in three zones. Producers use acks=all with idempotence; topics require two in-sync replicas. Consumers form groups, each partition is read by one member of a group, and offsets are committed after processing. Events are retained for 7 days for replay; compacted topics hold the latest state per key. Schemas are registered and checked for compatibility before a producer can publish. Failed messages go through delayed retry topics and finally a dead-letter topic owned by the consuming team.]

1.3 Decisions requested from reviewers

ID	Question	Recommended option	Section
Q-01	Example: Default delivery guarantee for consumers?	At-least-once with idempotent consumers; transactions only for log-to-log pipelines	7.6
Q-02	Example: Durability settings?	Replication factor 3, `min.insync.replicas=2`, `acks=all`, unclean election off	7.8
Q-03	Example: Retention for business topics?	7 days delete plus compacted state topics	7.2
Q-04	Example: Run ourselves or use a managed service?	[Decision]	10
[Q-nn]	[Question]	[Option]	[Section]

2. Requirements

2.1 Functional requirements

ID	Requirement	Priority
FR-01	Example: Publish keyed messages to named topics with headers	Must
FR-02	Example: Ordered delivery per key	Must
FR-03	Example: Several independent consumer groups per topic, each with its own position	Must
FR-04	Example: Replay from an offset or timestamp within retention	Must
FR-05	Example: Compacted topics keeping the latest value per key	Should
FR-06	Example: Retry with delay and dead-letter handling per consumer group	Must
FR-07	Example: Schema registration and compatibility checks	Must
FR-08	Example: Work-queue consumption with per-message acknowledgement	Could
[FR-nn]	[Requirement]	[Must / Should / Could]

2.2 Non-functional requirements

ID	Quality	Target	Measured as
NFR-01	Throughput	Example: 1 million messages/s (1 GB/s) at peak	Load test
NFR-02	Publish latency	Example: p99 at or below 30 ms with `acks=all`	Producer metrics
NFR-03	End-to-end latency	Example: p99 at or below 200 ms from publish to consumer poll	Timestamp headers
NFR-04	Durability	Example: No acknowledged message lost with one broker or one zone lost	Fault injection with verification
NFR-05	Availability	Example: Writes available with one zone lost	Zone failure game day
NFR-06	Recovery	Example: A consumer group recovers a 30-minute backlog within 15 min	Lag drain test
[NFR-nn]	[Quality]	[Target]	[Measurement]

2.3 Out of scope

[Example: cross-region active-active replication (later phase, mirrored topics)]
[Example: stream processing frameworks; teams choose their own]
[Example: messages larger than 1 MB (use object storage and send a reference)]
[Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

Input	Symbol	Value	Source
Peak messages per second (all topics)	$M_{p}$	Example: 1 000 000	Sale-day traffic x growth
Average messages per second	$M_{a}$	Example: 400 000	Daily curve
Average message size on the wire, compressed, with overhead	$m$	Example: 1 KB	Producer metrics
Replication factor	$f$	Example: 3	7.8
Consumer groups reading all data	$g$	Example: 3	Consumer inventory
Retention	$t$	Example: 7 days = 604 800 s	7.2
Usable disk per broker at 70% fill (24 TB raw)	$d$	Example: 16.8 TB	Hardware standard
Throughput per partition, producer side	$p_{w}$	Example: 10 MB/s	Benchmark
Throughput per partition, slowest consumer	$p_{c}$	Example: 5 MB/s (5 000 messages/s)	Consumer benchmark

3.2 Throughput and network

T_{peak} = M_{p} \times m = 10^{6} \times 1 \ \text{KB} = 1 \ \text{GB/s}, \qquad T_{avg} = 4 \times 10^{5} \times 1 \ \text{KB} = 400 \ \text{MB/s}

At peak, brokers receive producer traffic plus follower replication, and send consumer traffic plus replication:

B_{in} = T_{peak} \times f = 3 \ \text{GB/s}, \qquad B_{out} = T_{peak} \times g + T_{peak} \times (f - 1) = 3 + 2 = 5 \ \text{GB/s}

3.3 Storage and broker count

S = T_{avg} \times t \times f = 400 \ \text{MB/s} \times 604\,800 \ \text{s} \times 3 \approx 725.8 \ \text{TB}

n = \left\lceil \frac{725.8}{16.8} \right\rceil = \lceil 43.2 \rceil = 44 \Rightarrow 45 \ \text{brokers (15 per zone)}

Per broker at peak this is about $3\,000 / 45 \approx 67$ MB/s in and $5\,000 / 45 \approx 111$ MB/s out, well under a 25 Gbit/s (about 3.1 GB/s) network interface. Storage, not network, sets the broker count; tiered storage (7.2) would change that.

3.4 Partitions

The partition count must let both producers and the slowest consumer group keep up, with headroom $h$ for growth:

P = \max\left(\frac{T_{peak}}{p_{w}}, \frac{T_{peak}}{p_{c}}\right) \times h = \max(100, 200) \times 1.5 = 300

With replication factor 3 that is 900 partition replicas, 20 per broker. Apply the same formula per topic with that topic's own peak.

3.5 Backlog recovery

A consumer group with $P$ members at $5\,000$ messages/s each processes $1.5 \times 10^{6}$ messages/s. After a 30-minute outage at average load the backlog is $L = 4 \times 10^{5} \times 1\,800 = 7.2 \times 10^{8}$ messages, and with new messages still arriving at $\lambda = 4 \times 10^{5}$ /s:

t_{drain} = \frac{L}{C - \lambda} = \frac{7.2 \times 10^{8}}{1.5 \times 10^{6} - 4 \times 10^{5}} \approx 655 \ \text{s} \approx 11 \ \text{min}

This meets NFR-06. Catch-up reads come from disk rather than page cache, so schedule the drain test on production-sized brokers.

3.6 Summary

Quantity	Estimate	Design consequence
Ingress	1 GB/s peak, 400 MB/s average	Batching and compression at producers
Broker network	3 GB/s in, 5 GB/s out cluster-wide	About 67 MB/s in and 111 MB/s out per broker
Storage	about 726 TB for 7 days with 3 replicas	45 brokers, or fewer with tiered storage
Partitions	300 across business topics	20 replicas per broker
Backlog drain	about 11 min for 30 min of lag	Consumers sized at 1.5 times peak partition need

4. API design

4.1 Operations

Operation	Caller	Semantics
`produce(topic, key, value, headers)`	Producer	Appends to the partition chosen by key hash; returns partition and offset once acknowledged per `acks`
`fetch(topic, partition, offset, max_bytes)`	Consumer	Pull-based read from an offset; long-polls until data or timeout
`commit(group, partition, offset)`	Consumer	Records the next offset to read for the group
`begin`, `send_offsets`, `commit_txn`, `abort_txn`	Transactional producer	Atomic writes across partitions plus consumer offsets
`create_topic(name, partitions, config)`	Platform API	Through a reviewed request, not ad hoc

4.2 Producer and consumer configuration

The defaults below are from the Apache Kafka 4.3 configuration reference; check the documentation for your version.

Setting	Kafka 4.3 default	Platform setting	Why
`acks`	`all`	`all`	Acknowledge only when all in-sync replicas have the record
`enable.idempotence`	`true` (if no conflicting settings)	`true`	Broker drops duplicate retries using producer ID and sequence numbers
`max.in.flight.requests.per.connection`	5	5	Highest value that keeps ordering with idempotence
`linger.ms`	5	10 (Example)	Larger batches, better compression
`batch.size`	16 384 bytes	131 072 (Example)	Fewer requests at high throughput
`compression.type`	`none`	`zstd` or `lz4` (Example)	Cuts disk and network
Consumer `enable.auto.commit`	`true`	`false`	Commit after processing, not on a timer
Consumer `isolation.level`	`read_uncommitted`	`read_committed` for transactional topics	Hide aborted transactional records

4.3 Message envelope

{"key": "order-7f3a91", "headers": {"event_type": "OrderPlaced", "schema_id": "1042", "event_id": "0b6e6c8e-6f0e-4b0a-9e1d-2f3c4d5e6f70", "produced_at": "YYYY-MM-DDTHH:MM:SSZ", "trace_parent": "00-[trace-id]-[span-id]-01"}, "value": {"order_id": "7f3a91", "customer_id": "c-2231", "total_minor": 4599, "currency": "EUR"}}

event_id is the consumer's idempotency key; the value is serialised with the registered schema (7.11).

5. Data model

5.1 Entities

A topic has partitions; each partition replica is a sequence of segments on one broker; groups track offsets per partition.

5.2 Topic contract

Topic	Key	Partitions	Cleanup	Retention	Schema	Owner	Consumers
`orders.events.v1` (Example)	`order_id`	60	delete	7 days	`OrderEvent` (Avro)	Orders	Payments, fulfilment, analytics
`customers.state.v1` (Example)	`customer_id`	30	compact	Latest per key	`Customer` (Protobuf)	Customers	Search, CRM sync
`payments.retry.5m` (Example)	Original key	12	delete	1 day	Same as source	Payments	Payments consumer
`payments.dlq` (Example)	Original key	6	delete	30 days	Same as source plus error headers	Payments	Payments on-call
[Topic]	[Key]	[n]	[Policy]	[Retention]	[Schema]	[Team]	[Groups]

Topic names carry a major version; a breaking schema change creates a new topic and a migration window, never an in-place break.

6. High-level design

Component	Responsibility	Scales by
Brokers	Store partition replicas, serve produce and fetch	Brokers and disks
Controller quorum	Metadata, leader election, ISR changes (KRaft; Kafka 4.0 and later have no ZooKeeper mode)	Three or five controllers
Schema registry	Store schemas, enforce compatibility	Small replicated service
Consumer groups	Process partitions; one member per partition per group	Members up to partition count
Retry and dead-letter topics	Delay and park failed messages	Per consumer group
Platform API	Topic creation, ACLs, quotas, ownership	Single service

7. Detailed design

7.1 Topics and partitions

A topic is a named stream; a partition is an ordered, append-only sequence in which each record gets a monotonically increasing offset. Partitions are the unit of parallelism, ordering, replication and placement. The producer chooses the partition by hashing the key; records without a key are spread across partitions in batches.

7.2 Log segments and retention

Each partition replica is a directory of segment files, each with an index by offset and an index by timestamp. Only the newest (active) segment is written; older segments are immutable. Retention deletes whole closed segments, which is why it is cheap and why data can live slightly longer than the configured retention.

Time and size retention: Kafka 4.3 defaults to 168 hours (log.retention.hours) and rolls segments at 1 GiB (log.segment.bytes). At the example average of about 1.33 MB/s per partition (400 MB/s over 300 partitions), a segment fills in about $1\,073\,741\,824 / (1.33 \times 10^{6}) \approx 805$ s, so retention granularity is fine.
Compaction: with cleanup.policy=compact the log keeps at least the latest value per key; a null value is a tombstone that is removed after delete.retention.ms (default 24 hours). Use it for state topics, not for event histories.
Tiered storage: recent Kafka releases can offload closed segments to object storage, keeping only recent data on broker disks. It cuts the storage-driven broker count in section 3.3; check the feature status and limitations for your version before relying on it.

7.3 Produce path

One batch from an idempotent producer with acks=all, including the retry case:

7.4 Producers: acknowledgements and idempotence

`acks`	Acknowledged when	Loss risk	Latency
`0`	Sent, no broker response	Any failure loses data	Lowest
`1`	Leader has written the record	Loss if the leader fails before followers copy it	Low
`all`	All current in-sync replicas have the record	None while at least `min.insync.replicas` copies exist	Highest

Idempotence (available since Kafka 0.11, on by default in current releases) gives each producer an ID and numbers each batch per partition, so the leader discards duplicates from retries. It removes duplicates caused by the producer's own retries within a producer session; it does not deduplicate an application that sends the same business event twice, which is why events also carry event_id.

7.5 Consumer groups and offset management

Members of a group divide the partitions of the subscribed topics; each partition has exactly one member at a time. Committed offsets are stored as records in an internal compacted topic (__consumer_offsets in Kafka), so they are replicated like data. Kafka 4.0 made the new server-driven rebalance protocol (KIP-848) generally available, which reduces stop-the-world rebalances; Java consumers opt in with group.protocol=consumer, and other clients may lag behind, so check the client support for your language.

Commit rules for the platform:

Disable auto commit; commit after the processing side effects are durable.
Commit the offset of the next record to read, per partition.
On rebalance, finish or abandon in-flight work and commit before releasing partitions.
Consumers must tolerate seeing a record again after a crash (7.6).

For work-queue use where many consumers share a partition with per-message acknowledgement, Kafka share groups (KIP-932) were declared production-ready in Kafka 4.2; confirm client support before choosing them over classic consumer groups.

7.6 Delivery semantics and their real limits

Semantics	Mechanism	What it guarantees	What it does not
At-most-once	Commit offset before processing	No duplicates	Messages lost if the consumer crashes mid-processing
At-least-once	Commit after processing	No loss within retention	Duplicates after crashes, rebalances or retries
Exactly-once within the log	Idempotent producer plus transactions; consumers in `read_committed`; offsets committed in the same transaction	A read-process-write pipeline from topic to topic produces each output once, with its input offset	Any side effect outside the log: database writes, HTTP calls, emails
Effectively-once end to end	At-least-once plus idempotent processing (dedupe on `event_id`, upserts, conditional writes) or a transactional outbox	Each business effect applied once	Requires discipline in every consumer

The platform default is at-least-once with idempotent consumers. Transactions are used for stream processing jobs that read and write topics (for example Kafka Streams with exactly-once processing enabled). Transactional producers need a stable transactional.id so that a restarted instance fences the old one; a transaction timeout aborts work from a producer that disappeared.

7.7 Ordering guarantees

Order is guaranteed only within a partition. Records with the same key go to the same partition while the partition count stays the same.
With idempotence and up to 5 in-flight requests, retries do not reorder records within a partition.
Increasing the partition count changes the key-to-partition mapping for new records, so per-key order breaks across the change. Over-provision partitions at creation (3.4) or create a new topic and migrate.
A consumer that processes records of one partition in parallel threads must keep per-key order itself (for example, one queue per key hash).
Retry topics break order for the failed key; topics that require strict per-key order pause the partition instead of forwarding to retry.

7.8 Replication, in-sync replicas and leader election

Each partition has one leader and followers that fetch from it. The leader tracks the in-sync replica set (ISR): replicas that keep an active session with the controller and are not too far behind (Kafka 4.3 default for replica.lag.time.max.ms is 30 s). A record is committed when all ISR members have it, and consumers only read up to the high watermark.

Setting	Kafka 4.3 default	Platform setting	Effect
`default.replication.factor`	1	3	One replica per zone
`min.insync.replicas`	1	2	With `acks=all`, writes fail if fewer than 2 replicas are in sync
`unclean.leader.election.enable`	`false`	`false`	Never elect an out-of-sync replica, which could lose committed data
`offsets.topic.replication.factor`	3	3	Offsets as durable as data

With replication factor 3 and min.insync.replicas=2, one broker or one zone can be lost without losing acknowledged data or write availability; a second simultaneous loss in the same partition stops writes (producers get a "not enough replicas" error) rather than risking loss. The controller quorum elects a new leader from the ISR when a leader fails.

7.9 Backpressure

The log decouples rates, but every buffer is finite:

Point	Mechanism	Platform setting
Producer	Bounded send buffer; `send` blocks then fails when full	Fail fast after a short block; shed or spill non-critical events
Broker	Produce and fetch byte-rate quotas per client or user; request throttling	Quotas per team from the capacity plan
Consumer	Pull model; bounded records per poll; pause and resume partitions	Pause when downstream is saturated; alert on lag
Downstream	Consumer rate limits to protect databases and third-party APIs	Token bucket per dependency

Consumer lag (latest offset minus committed offset) is the main signal. For timeouts, retry budgets and load-shedding policies across services, align with the Resilience and load shedding design (system-design/resilience-load-shedding-design).

7.10 Retries and dead-letter topics

Most log technologies have no built-in dead-letter topic for plain consumers, so the platform provides a library:

Classify the error: transient (timeout, 503) or permanent (validation, unknown schema).
Transient: retry in-process up to 3 times with jittered backoff, then publish to a delay topic (retry.5m, then retry.1h) with headers for the original topic, partition, offset, attempt and error.
Permanent, or attempts exhausted: publish to the group's dead-letter topic and commit the original offset.
The owning team triages the dead-letter topic with a replay tool that republishes selected records to the original topic or a retry topic.

Dead-letter topics count as production data: same access controls, retention of 30 days and an alert when the rate exceeds a threshold.

7.11 Schema registry and compatibility

Producers register Avro, Protobuf or JSON Schema definitions and send the schema ID in each message; consumers fetch schemas by ID and cache them. The registry rejects incompatible changes according to the topic's mode:

Mode	Allowed change	Upgrade order
Backward	New schema can read data written with the previous one (add optional fields, remove fields)	Consumers first
Forward	Previous schema can read data written with the new one	Producers first
Full	Both directions	Either
None	Anything	Not allowed for shared topics

The platform default is backward compatibility with transitive checks for shared topics. Document each topic's contract in the event API specification (architecture/event-api-specification).

8. Scalability and performance

Concern	Approach	Limit before redesign
Throughput	Add brokers and partitions; producer batching and compression	Example: per-broker network above 50% of NIC
Storage	7-day retention; tiered storage for longer replay	Disk fill above 70%
Partition count	Over-provision at creation; cap per broker	Example: several thousand replicas per broker
Hot partition	Better key choice, key salting for non-ordered data	One key above a partition's throughput
Consumer parallelism	Members up to partition count; per-key parallel processing inside members	Partition count

Latency budget for publish with acks=all (target p99 30 ms):

Step	Budget
Producer linger and batching	10 ms
Network to leader	1 ms
Append to page cache	1 ms
Follower fetch round trip across zones (two followers in parallel)	5 ms
Acknowledgement to producer	1 ms
Headroom (garbage collection, queueing, disk flush)	12 ms

9. Reliability, security and operations

9.1 Failure modes

Failure	Impact	Detection	Mitigation
Broker down	Leaders move; ISR shrinks	Under-replicated partitions	Leader election from ISR; replace broker
Zone down	One third of brokers gone; ISR at 2	Zone health, under-replicated partitions	`min.insync.replicas=2` keeps writes up; rack-aware placement
Slow follower	Leaves ISR; durability margin shrinks	ISR shrink rate	Investigate disk or network; throttle reassignments
Consumer bug writes bad data	Downstream corruption	Data quality checks	Reset group offsets to a timestamp and replay
Poison message	Consumer stuck on one offset	Lag growing on one partition	Retry and dead-letter path (7.10)
Controller quorum loss	No leader elections or metadata changes	Controller health	Five controllers across three zones; tested recovery runbook
[Failure]	[Impact]	[Signal]	[Mitigation]

9.2 Security

Threat	Control
Unauthorised produce or consume	mTLS or SASL authentication; ACLs per topic and group, owned by teams
Sensitive data in events	Data classification per topic; field-level encryption for regulated fields; no secrets in headers
Data retained too long	Retention per topic reviewed by data governance; compaction tombstones for erasure
Schema poisoning	Registry writes limited to producer service accounts; compatibility enforced
Noisy neighbour	Quotas per client and team
[Threat]	[Control]

9.3 Observability

Signal	Metric or log	Alert threshold
Under-replicated partitions	Count per cluster	Any for more than 5 min
Offline partitions	Count	Any
Consumer lag	Messages and seconds behind, per group	Lag in seconds above the group's SLO
Produce errors	Error rate by type	Above 0.1% for 5 min
Dead-letter rate	Messages per minute per group	Above the group threshold
Disk usage	Per broker	Above 70%

9.4 Cost estimate

Item	Unit cost	Quantity	Monthly cost
Brokers	[USD per broker-month]	Example: 45	[USD]
Controllers	[USD per node-month]	Example: 5	[USD]
Cross-zone replication traffic	[USD per GB]	$2 \times 400$ MB/s average	[USD]
Schema registry and platform API	[USD per month]	1 deployment	[USD]
Tiered object storage (if used)	[USD per TB-month]	[TB]	[USD]
Total			[USD]

10. Trade-offs and alternatives

Decision	Options	Chosen	Why	What we give up
Model	Broker-centric queue (per-message ack, deleted on consume); partitioned log	Partitioned log	Replay, many independent consumers, high throughput	Per-message routing features; work-queue semantics need share groups
Delivery default	At-most-once; at-least-once; exactly-once everywhere	At-least-once plus idempotent consumers	Works with external side effects	Every consumer must deduplicate
Durability	`acks=1`; `acks=all` with `min.insync.replicas=2`	`acks=all`, ISR 2	No loss on one failure	Higher publish latency
Retention	Short; long on brokers; tiered	7 days on brokers, tiered later	Replay window for incidents	Disk cost drives broker count
Operation	Self-managed; managed service	[Choice]	[Reason]	[Cost]
[Decision]	[Options]	[Chosen]	[Why]	[Cost]

11. Delivery plan and risks

11.1 Milestones

Milestone	Scope	Exit criterion	Target date
M1	Cluster, platform API, schema registry, one pilot topic	Fault injection suite passes; no acknowledged loss	YYYY-MM-DD
M2	Retry and dead-letter library, consumer lag dashboards	Poison-message test recovers without manual offset edits	YYYY-MM-DD
M3	Orders and payments topics in production	NFR-02 and NFR-03 met for 14 days	YYYY-MM-DD
M4	Teams' legacy queues migrated	Legacy queues decommissioned	YYYY-MM-DD

11.2 Rollout and rollback

[Example: Producers dual-publish to the legacy queue and the new topic; consumers move one group at a time with a flag. Rollback flips the consumer flag back; dual publishing continues until 30 days after the last consumer moved.]

11.3 Risks

ID	Risk	Likelihood	Impact	Mitigation	Owner
R-01	Example: Consumers assume exactly-once and double-apply effects	High	High	Idempotency checklist in consumer reviews; dedupe library	[@name]
R-02	Example: Partition count too low, re-keying needed	Medium	High	Over-provision per 3.4; capacity review each quarter	[@name]
R-03	Example: Schema change breaks a consumer	Medium	Medium	Registry compatibility checks in CI	[@name]
[R-nn]	[Risk]	[L / M / H]	[L / M / H]	[Mitigation]	[@name]

12. Appendices

12.1 Open questions

ID	Question	Owner	Due
OQ-01	Example: Which topics need cross-region mirroring, and with what lag?	[@name]	YYYY-MM-DD
OQ-02	Example: Adopt share groups for the job-processing workload?	[@name]	YYYY-MM-DD
[OQ-nn]	[Question]	[@name]	YYYY-MM-DD

12.2 Glossary

Term	Meaning
Partition	Ordered, append-only log that is the unit of parallelism and replication
Offset	Position of a record within a partition
Segment	One file of a partition's log; retention deletes whole segments
ISR	In-sync replicas: replicas caught up closely enough to the leader
High watermark	Highest offset replicated to all ISR members; consumers read up to it
Consumer group	Set of consumers that share the partitions of a subscription
Consumer lag	Difference between the latest offset and the group's committed offset
Dead-letter topic	Topic that holds messages a consumer could not process
[Term]	[Meaning]

12.3 References

Reference	Use
Apache Kafka 4.3 documentation: design, producer, consumer and broker configuration	Concepts and default values in 4.2, 7.2, 7.4, 7.6 and 7.8
J. Kreps, N. Narkhede, J. Rao, "Kafka: a Distributed Messaging System for Log Processing", NetDB 2011	Log-based design, pull consumers, partitioned topics
Apache Kafka 4.0 and 4.2 release announcements	KRaft-only operation, KIP-848 rebalance protocol, share groups (KIP-932)
Event API specification (`architecture/event-api-specification`)	Topic contracts and schemas
Capacity estimation worksheet (`system-design/back-of-envelope-estimation`)	Method for section 3
Capacity plan (`ops/capacity-plan`)	Ongoing broker and partition capacity

Standard and licence note

Based on the Apache Kafka 4.3 documentation and the Kafka paper by Kreps, Narkhede and Rao (2011), used as concept sources, and on common industry practice for event streaming platforms. Default values are cited as published for Kafka 4.3 in October 2026; check the documentation for the version you run, because defaults change between releases. This is a reference design to adapt, with illustrative numbers. Original text written for this collection; no documentation text is reproduced. Apache Kafka is a trademark of the Apache Software Foundation. Template licence: MIT.

Modèles associés

Conception système