Streaming Solution Design: [Solution name]
Solution design for a real-time data solution built on an event
streaming platform and a stream processor (for example Apache Kafka and
Apache Flink): use cases and latency objectives, sources and change
data capture, event contracts and schema governance, topic design, the
processing topology, time, windows and state, delivery guarantees
(exactly-once and idempotency), sinks, capacity, scaling, reliability,
security, lag monitoring and testing. Event contracts are documented
with the AsyncAPI Specification 3.1.0. Replace every [bracketed]
value and every row marked Example. Worked examples follow a
retailer that turns store and online sales events into live stock
availability and payment fraud alerts.
Document control
| Field | Value |
|---|
| Document ID | [STR-001] |
| Solution name | [Name] |
| Streaming platform | [Apache Kafka 4.x managed service / self-managed cluster / other], version [x.y] |
| Stream processor | [Apache Flink 2.x / Kafka Streams / other], version [x.y] |
| Schema registry | [Product and version] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@streaming-architect] |
| Approvers/reviewers | [@platform-owner], [@security-architect], [@data-platform-architect], [@sre-lead] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal / Confidential] |
Approvals
| Role | Name | Decision | Date |
|---|
| Business owner | [@name] | [Approve / Approve with conditions / Reject] | YYYY-MM-DD |
| Streaming platform owner | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| SRE | [@name] | [Decision] | YYYY-MM-DD |
Revision history
| Version | Date | Author | Change summary |
|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 1.0 | YYYY-MM-DD | [@name] | Example: Stock availability stream approved |
| 1.1 | YYYY-MM-DD | [@name] | Example: Added fraud scoring job and alert sink |
Contents
1. Use cases and latency objectives
| ID | Use case | Consumer | Latency objective (p95) | Value of lower latency | Batch alternative rejected because |
|---|
| SUC-001 (Example) | Live stock availability per product and store | Web shop, store app | 30 seconds | Fewer orders for out-of-stock items | Hourly batch caused 2 % order cancellations |
| SUC-002 (Example) | Card-not-present fraud alert | Payment risk team, order service | 2 seconds | Order held before shipment | Fraud detected after dispatch |
| SUC-003 (Example) | Sales events to the lakehouse bronze layer | Data platform | 5 minutes | Intraday reporting | Hourly extracts load the POS database |
| [SUC-NNN] | [Use case] | [Consumer] | [Latency] | [Value] | [Reason] |
Record the objectives in the SLO document (dev/slo-doc) with the
measurement point at each end.
2. Requirements and delivery guarantees
2.1 Requirements
| ID | Requirement | Type | Acceptance criterion |
|---|
| SRQ-001 (Example) | Compute available stock as on-hand minus reserved, per product and store | Functional | Matches the stock system snapshot within 0.5 % at daily reconciliation |
| SRQ-002 (Example) | Score each online payment with rules and a model | Functional | Score attached to 100 % of payment events |
| SRQ-003 (Example) | Handle peak of 50,000 events per second | Performance | Load test at 1.5 times peak without lag growth |
| SRQ-004 (Example) | No lost sales event after a broker or job failure | Reliability | Chaos test with zero missing events against source counts |
| SRQ-005 (Example) | Personal data encrypted in transit and access-controlled per topic | Security | Audit of ACLs and TLS configuration |
| [SRQ-NNN] | [Requirement] | [Type] | [Criterion] |
2.2 Delivery guarantee per output
| Output | Guarantee | How it is achieved | Reason |
|---|
| stock.availability topic (Example) | Effectively-once | Transactional sink committed on checkpoint; consumers read committed only | Downstream sums would drift with duplicates |
| Fraud alert to order service (Example) | At-least-once with idempotent receiver | Alert carries a deterministic identifier; receiver ignores repeats | 2-second objective rules out waiting for checkpoints |
| Bronze tables (Example) | Effectively-once | Sink commits files and table snapshot on checkpoint | Replays must not duplicate rows |
| [Output] | [Guarantee] | [Mechanism] | [Reason] |
3. Sources and change data capture
| ID | Source | Method | Event types | Key | Rate (peak) | Owner |
|---|
| ESR-001 (Example) | POS gateway | Native producer, idempotent | SaleRecorded, ReturnRecorded | store_id | 30,000 per second | @retail-ops |
| ESR-002 (Example) | E-commerce order service | Transactional outbox read by CDC | OrderPlaced, OrderCancelled, PaymentAuthorised | order_id | 8,000 per second | @ecom-lead |
| ESR-003 (Example) | Warehouse system | Log-based CDC | StockAdjusted, StockReserved | product_id plus location_id | 12,000 per second | @supply-chain |
| [ESR-NNN] | [Source] | [Method] | [Types] | [Key] | [Rate] | [@owner] |
Example peak total: 30,000 + 8,000 + 12,000 = 50,000 events per
second. Section 10 sizes the platform from this figure. Snapshot,
ordering and schema change handling for CDC follow CDC Pipeline Design
(system-design/design-cdc-pipeline).
4. Event contracts and schema governance
4.1 Contract rules
- Describe each channel and message in an AsyncAPI 3.1.0 document
(
asyncapi: 3.1.0, with info, servers, channels, operations
and components), stored with the producer's code and published to
the catalogue. Detail it in Event API Specification
(architecture/event-api-specification).
- Register payload schemas (Avro, Protobuf or JSON Schema) in the schema
registry; producers fail fast if the schema is not registered.
- Use the envelope attributes of CloudEvents 1.0.2 (
id, source,
type, specversion, time) where events cross organisational
boundaries, so routing does not depend on the payload.
- Include an event identifier, the business event time and the producer
version in every event.
- Treat the topic, key, schema and guarantee of a published stream as a
data contract (
ai-data/data-contract).
4.2 Compatibility policy
| Stream class | Compatibility mode (Example) | Allowed change | Breaking change process |
|---|
| Internal to one team | Backward | Add fields with defaults, delete fields | New topic version, dual publish for [2] weeks |
| Shared across domains | Full transitive | Add optional fields with defaults only | New major contract version, consumer sign-off, [90] days dual publish |
| External partners | Full transitive plus AsyncAPI review | As above | Partner notice of [6] months |
Compatibility mode names follow common schema registry practice
(backward, forward, full, each optionally transitive, or none); check the
modes your registry supports.
5. Topic design
| Topic | Key | Partitions | Retention | Cleanup | Replication | Producers | Consumers |
|---|
| retail.sales.v1 (Example) | store_id | 24 | 7 days | Delete | 3, min in-sync 2 | POS gateway | Stock job, bronze sink |
| ecom.orders.v1 (Example) | order_id | 12 | 7 days | Delete | 3, min in-sync 2 | Outbox CDC | Fraud job, stock job, bronze sink |
| wms.stock-changes.v1 (Example) | product_id:location_id | 24 | 7 days | Delete | 3, min in-sync 2 | WMS CDC | Stock job, bronze sink |
| stock.availability.v1 (Example) | product_id:store_id | 24 | Latest value per key | Compact | 3, min in-sync 2 | Stock job | Web shop, store app |
| fraud.alerts.v1 (Example) | order_id | 6 | 30 days | Delete | 3, min in-sync 2 | Fraud job | Order service, risk team |
| [domain.entity.vN] | [Key] | [Count] | [Retention] | [Policy] | [RF] | [Producers] | [Consumers] |
Naming convention: <domain>.<entity-or-event>.<major version>. Where
an Event-Driven Architecture Design (architecture/event-driven-architecture-design)
exists, its convention (section 5.1, with environment and kind parts)
takes precedence; the short names here keep the examples readable. Keep a
dead-letter topic per consuming job (<source-topic>.dlq.<job>, the same
rule as the EDA design) with the original
payload, the error and the source offset.
Example skew check: store_id has 1,800 values over 24 partitions, about
75 stores per partition. The largest store produces [3] times the average
volume, so verify the hottest partition stays under the per-partition
limit measured in section 10.
6. Processing topology
| Job | Operators with state | Parallelism (Example) | Deployment | Owner |
|---|
| stock-availability (Example) | Availability per product and store, deduplication by event id | 24 | Application mode on Kubernetes, one job per use case | @retail-data |
| fraud-scoring (Example) | Per-customer windows, model in memory | 12 | Application mode, separate cluster | @risk-data |
| [Job] | [Stateful operators] | [Parallelism] | [Deployment] | [@owner] |
7. Time, windows and state
7.1 Time and lateness
| Stream | Time used | Watermark strategy | Allowed lateness | Late events after lateness |
|---|
| retail.sales.v1 (Example) | Event time (till timestamp) | Bounded out-of-orderness of 30 seconds; idle source detection after 1 minute | 10 minutes | Side output to a correction topic, applied in the lakehouse |
| ecom.orders.v1 (Example) | Event time | Bounded out-of-orderness of 5 seconds | 1 minute | Scored without window features, flagged |
| [Stream] | [Time] | [Strategy] | [Lateness] | [Handling] |
A watermark at time t asserts that no further events at or before t are
expected. Watermarks advance per parallel source instance, and an
operator with several inputs follows the slowest; an idle partition can
therefore stall results, which is why idle source detection matters.
7.2 Windows
| ID | Window | Type | Size | Trigger | Use |
|---|
| WIN-001 (Example) | Customer payment features | Sliding | 10 minutes, every 1 minute | On watermark | Fraud score |
| WIN-002 (Example) | Store sales per minute | Tumbling | 1 minute | On watermark plus early firing every 10 seconds | Store dashboard |
| WIN-003 (Example) | Browsing session | Session | Gap of 30 minutes | On gap | Basket abandonment |
| [WIN-NNN] | [Window] | [Type] | [Size] | [Trigger] | [Use] |
7.3 State
| Operator | State content | Keys | Size per key | Total state | Time-to-live | Backend |
|---|
| Availability (Example) | On-hand, reserved, last event id | 2,000,000 product-store pairs | 2 KB | 4 GB | None (live entities) | Embedded key-value store on local disk, incremental checkpoints |
| Deduplication (Example) | Event ids seen | Events in last 15 minutes | 64 bytes | Section 10.4 | 15 minutes | As above |
| Customer windows (Example) | Payment amounts and counts | Active customers | 1 KB | [Size] | 24 hours | As above |
| [Operator] | [Content] | [Keys] | [Size] | [Total] | [TTL] | [Backend] |
Set a time-to-live on every state that is not a live entity; state that
grows without bound is the most common cause of slow checkpoints and
long recovery.
8. Exactly-once and idempotency
| Layer | Mechanism (Example) | Configuration to check |
|---|
| Producers | Idempotent producer with acknowledgement from all in-sync replicas | acks set to all, idempotence enabled, retries enabled |
| Processor state | Consistent checkpoints of all operator state and source offsets | Checkpoint interval [60 s], minimum pause, timeout, retained checkpoints |
| Broker sink | Transactional writes committed when a checkpoint completes | Unique transactional id prefix per job; transaction timeout longer than the longest checkpoint plus restart time |
| Downstream consumers | Read committed data only | Consumer isolation level read committed |
| External database sink | Idempotent upsert keyed by business key plus event version | Upsert ignores versions older than stored |
| Alerts | Deterministic alert id derived from order id and rule id | Receiver stores processed ids for [7] days |
Consequence of the transactional sink: results become visible to
read-committed consumers only after the checkpoint completes. With a
60-second interval the stock topic can lag by up to about a minute plus
commit time, which breaks the 30-second objective in SUC-001. Example
choice: checkpoint the stock job every 10 seconds (state is about 6.9 GB, section 10.4, and
checkpoints are incremental) and keep the fraud alert path out of the
transaction, as stated in section 2.2. Record the decision in the
Architecture Decision Log (architecture/architecture-decision-log).
9. Sinks and serving
| ID | Sink | Data | Write pattern | Guarantee | Consumer |
|---|
| SNK-001 (Example) | Compacted topic stock.availability.v1 | Latest availability per key | Upsert by key | Effectively-once | Web shop cache, store app |
| SNK-002 (Example) | Key-value cache | Availability for page rendering | Upsert from SNK-001 by a consumer service | At-least-once, idempotent | Web shop |
| SNK-003 (Example) | Lakehouse bronze table | All sales, order and stock events | Append, commit on checkpoint, compaction daily | Effectively-once | Data platform |
| SNK-004 (Example) | fraud.alerts.v1 and order service API | Alerts | Publish then call with retries | At-least-once, idempotent | Order service |
| [SNK-NNN] | [Sink] | [Data] | [Pattern] | [Guarantee] | [Consumer] |
Streaming sinks into the lakehouse produce many small files; schedule
compaction as described in Data Platform Architecture
(architecture/data-platform-architecture).
10. Capacity estimation
Units in this section are decimal: 1 KB = 1,000 bytes, 1 MB = 10^6
bytes, 1 TB = 10^12 bytes. All figures are Example values.
10.1 Throughput
| Step | Calculation | Result |
|---|
| Peak ingress | 50,000 events per second x 1 KB average | 50 MB/s |
| Replicated write at replication factor 3 | 50 MB/s x 3 | 150 MB/s across brokers |
| Consumer egress (sales read by two groups, orders by three, stock changes by two; section 5) | 30 MB/s x 2 + 8 MB/s x 3 + 12 MB/s x 2 | 108 MB/s |
Consequence: brokers must sustain 150 MB/s of disk writes plus 100 MB/s
of replication traffic between brokers (two follower copies of 50 MB/s)
and about 108 MB/s of consumer reads, mostly served from page cache;
size network and disks with [50 %] headroom.
10.2 Retention storage
| Step | Calculation | Result |
|---|
| Average daily volume | 20,000 events per second x 1 KB x 86,400 s | 1.728 TB per day |
| Seven days logical | 1.728 TB x 7 | 12.096 TB |
| With replication factor 3 | 12.096 TB x 3 | 36.288 TB |
Consequence: about 36 TB of broker storage for 7 days. If the platform
offers tiered storage, keep [1] day on local disk and the rest in object
storage, which cuts local disk to about 1.728 x 3 = 5.2 TB.
10.3 Partitions
| Step | Calculation | Result |
|---|
| Per-partition throughput measured in a load test | Given | 5 MB/s |
| Peak on retail.sales.v1 (POS stream only, ESR-001) | 30,000 events per second x 1 KB | 30 MB/s |
| Minimum partitions for peak | 30 MB/s / 5 MB/s | 6 |
| Minimum partitions for twice today's peak | 60 MB/s / 5 MB/s | 12 |
| Chosen for retail.sales.v1 | Three-year growth and stock job parallelism; divides evenly by 8, 12 and 24 | 24 |
Consequence: 24 partitions cap the parallelism of any consumer group at
24. Increasing partitions later changes key-to-partition mapping and
breaks per-key ordering for in-flight data, so choose the count for the
three-year growth now.
10.4 State
| Step | Calculation | Result |
|---|
| Availability state | 2,000,000 keys x 2 KB | 4 GB |
| State per subtask at parallelism 24 | 4 GB / 24 | about 167 MB |
| Deduplication state | 50,000 events per second x 900 s x 64 bytes | 2.88 GB |
| Total stock job state | 4 GB + 2.88 GB | about 6.9 GB |
Consequence: about 290 MB of state per subtask (6.9 GB / 24) restores in
seconds from local disk and in under a minute from object storage at
typical throughput; 10-second incremental checkpoints are feasible. If
deduplication by event id were held for 24 hours instead of 15 minutes,
it would grow 96 times to about 276 GB, so keep the window short and rely
on idempotent producers for older duplicates.
11. Scaling and performance
| Component | Scaling method | Limit | Trigger (Example) |
|---|
| Brokers | Add brokers and reassign partitions | Disk and network per broker | Disk above 70 % or network above 60 % |
| Stream jobs | Rescale parallelism from a savepoint, or autoscaler | Partition count of the input topic | Consumer lag growing for 5 minutes |
| Sinks | Scale the receiving store; batch writes | Write capacity of the store | Backpressure on sink operator above 50 % |
| Schema registry | Replicas behind a load balancer | Rarely the bottleneck | Request latency above 50 ms |
Performance targets: p95 end-to-end latency per SUC objective, measured
by comparing event time with sink write time on sampled events. Plan load
tests in the Performance Test Plan (quality/performance-test-plan).
12. Reliability and recovery
| Failure (Example) | Effect | Detection | Recovery |
|---|
| One broker fails | Leaders move to other replicas; producers retry | Under-replicated partitions alert | Automatic; replace broker |
| Job task manager fails | Job restarts from last checkpoint; output replays to last commit | Restart counter, lag | Automatic within [60 s] |
| Poison message | Parsing fails | Dead-letter topic rate | Message routed to DLQ, job continues |
| Downstream cache unavailable | Consumer service lags | Lag on its consumer group | Retry with backoff, cache rebuilt from compacted topic |
| Region outage | Platform unavailable in the region | Health checks | Fail over to the standby region per resilience tier |
Recovery objectives: RPO [1 minute] and RTO [30 minutes] for the stock
stream (tier per Resilience Tiering and HA/DR Design,
architecture/resilience-tiering-ha-dr-design). Cross-region replication
copies topics asynchronously; consumer offsets must be translated, and
stateful jobs restart from the latest replicated savepoint or rebuild
state by replaying compacted topics. Take a savepoint before every job
upgrade and test restore quarterly.
13. Security and privacy
| Control | Design (Example) |
|---|
| Authentication | Mutual TLS or OAuth-based authentication for every client; one identity per producer and job |
| Authorisation | ACLs per topic and consumer group; producers write only their topics |
| Encryption | TLS in transit; encryption at rest on brokers and tiered storage |
| Personal data | No personal data in keys; customer identifiers pseudonymised in sales events; card data never on the stream (tokens only) |
| Retention and erasure | Short retention (7 days) on raw topics; compacted topics with personal data support deletion by key (tombstone) |
| Audit | Admin operations and ACL changes logged to the security log store |
| Threat analysis | Producer impersonation, topic flooding, data exfiltration through consumer groups; recorded in security/threat-model |
Assess personal data processing in the DPIA (compliance/dpia) when
events feed profiling such as fraud scoring.
14. Operations and lag monitoring
| Signal | Threshold (Example) | Action |
|---|
| Consumer lag in seconds (event time of last processed record versus now) | Above 2 times the latency objective for 5 minutes | Page on-call; check backpressure and sink health |
| Checkpoint duration | Above 50 % of the interval | Investigate state growth and storage |
| Checkpoint failures | 3 in a row | Page on-call |
| Job restarts | More than 3 per hour | Page on-call |
| Dead-letter rate | Above 0.1 % of input | Notify producer owner |
| Under-replicated partitions | Any for 5 minutes | Platform on-call |
| Schema registration rejected | Any | Notify producer owner |
Write signal definitions and dashboards in the Observability
Specification (ops/observability-spec) and runbooks for lag, failed
checkpoints, DLQ replay and savepoint restore in dev/runbook.
15. Testing
| ID | Test | Method | Pass criterion |
|---|
| STT-001 (Example) | Operator logic | Unit tests with a test harness that controls time | Expected outputs for in-order, out-of-order and late events |
| STT-002 (Example) | Contract compatibility | Schema check in CI against the registry | No incompatible change merged |
| STT-003 (Example) | Exactly-once under failure | Kill task managers and brokers during a replay of known data | Output equals expected with no duplicates or gaps |
| STT-004 (Example) | Load | Replay of a peak day at 1.5 times rate | Lag stable, latency objectives met |
| STT-005 (Example) | Reconciliation | Daily compare of stream totals with source system totals | Within 0.5 % (SRQ-001) |
| STT-006 (Example) | Upgrade | Savepoint, upgrade, restore in staging | State restored, no lost events |
| [STT-NNN] | [Test] | [Method] | [Criterion] |
16. Trade-offs, roadmap and risks
16.1 Trade-offs
| Decision | Chosen (Example) | Alternative | Why chosen | Revisit when |
|---|
| Processor | Flink-style engine with event time and managed state | Broker-native stream library | Several input topics, large state, event-time windows | Use cases become simple per-topic transforms |
| Stock output | Compacted topic plus cache | Direct writes to the cache | Replayable, decouples consumers | Cache supports change streams natively |
| Fraud path | At-least-once with idempotent receiver | Transactional sink | 2-second objective | Checkpoint interval can drop below 1 second |
| Hosting | Managed streaming service | Self-managed cluster | Operations effort | Cost at ten times volume exceeds a dedicated team |
16.2 Roadmap and risks
| Increment | Scope | Exit criterion |
|---|
| S1 (Example) | Sales and stock topics, bronze sink | Reconciliation within 0.5 % for 4 weeks |
| S2 (Example) | Stock availability job and cache | SUC-001 objective met in production |
| S3 (Example) | Fraud scoring | False positive rate agreed with risk team |
| ID | Risk | Likelihood | Impact | Response | Owner |
|---|
| RSK-001 (Example) | Hot partition from a flagship store | Medium | Medium | Key salting for that store or a finer key | @streaming-architect |
| RSK-002 (Example) | Schema change breaks consumers | Medium | High | Registry enforcement, STT-002 in CI | @platform-owner |
| RSK-003 (Example) | State growth slows checkpoints | Medium | High | TTL on all state, checkpoint alert | @sre-lead |
| [RSK-NNN] | [Risk] | [L / M / H] | [L / M / H] | [Response] | [@owner] |
17. Glossary and references
| Term | Meaning |
|---|
| Event time | Time at which the event happened at the source |
| Watermark | Marker asserting that no earlier events are expected, which lets windows close |
| Allowed lateness | Extra time during which late events still update a closed window |
| Checkpoint | Consistent snapshot of operator state and input positions used for recovery |
| Savepoint | User-triggered snapshot used for upgrades, rescaling and migration |
| Compacted topic | Topic that keeps at least the latest record per key |
| Consumer lag | How far a consumer is behind the end of a partition, in records or time |
| Effectively-once | Each input affects the results exactly once despite retries and restarts |
| Outbox pattern | Writing an event to a database table in the same transaction as the state change, then publishing it |
| Reference | Version or date checked | Use |
|---|
| AsyncAPI Specification, asyncapi.com | 3.1.0, released 2026-01-31, checked 2026-10-23 | Event contracts (section 4) |
| CloudEvents, cloudevents.io (CNCF) | 1.0.2 | Envelope attributes |
| Apache Kafka documentation and upgrade notes, kafka.apache.org | 4.3 line (4.3.1 released 2026-06-25); KRaft only since 4.0; share groups production-ready since 4.2; checked 2026-10-23 | Platform concepts |
| Apache Flink documentation, flink.apache.org | 2.3.0 (2026-06-25); 1.20 is the LTS line; checked 2026-10-23 | Time, watermarks, windows, exactly-once Kafka sink |
| Schema registry documentation (for example Confluent Schema Registry, docs.confluent.io) | Checked 2026-10-23 | Compatibility mode names |
| [Managed service documentation] | [Version] | [Limits and quotas] |
Standard and licence note
Based on the AsyncAPI Specification 3.1.0 (Linux Foundation, Apache
License 2.0) for event contracts and CloudEvents 1.0.2 (CNCF) for event
envelopes, with concepts described from the public Apache Kafka and
Apache Flink documentation (Apache License 2.0). The text and the worked
design are original and no documentation text is reproduced. This
template is original text written for this collection. Check the current
official versions of each specification and product before use. Apache
Kafka and Apache Flink are trademarks of the Apache Software Foundation,
and Confluent is a trademark of its owner; this template is not
affiliated with or endorsed by them. Template licence: MIT.