# Streaming Solution Design: [Solution name]

> Solution design for a real-time data solution built on an event
> streaming platform and a stream processor (for example Apache Kafka and
> Apache Flink): use cases and latency objectives, sources and change
> data capture, event contracts and schema governance, topic design, the
> processing topology, time, windows and state, delivery guarantees
> (exactly-once and idempotency), sinks, capacity, scaling, reliability,
> security, lag monitoring and testing. Event contracts are documented
> with the AsyncAPI Specification 3.1.0. Replace every `[bracketed]`
> value and every row marked *Example*. Worked examples follow a
> retailer that turns store and online sales events into live stock
> availability and payment fraud alerts.

<!--
Guidance: How to use this template
- This is a solution design that uses an existing streaming platform and
  processing engine. To design the engine itself, use Stream Processing
  Engine Design (`system-design/design-stream-processing-engine`); for
  the capture of database changes in depth, use CDC Pipeline Design
  (`system-design/design-cdc-pipeline`).
- Start with latency objectives and delivery guarantees (sections 1 and
  2). They decide checkpoint intervals, sink types and cost.
- Every event type in section 4 needs an AsyncAPI document in Event API
  Specification (`architecture/event-api-specification`) before a
  producer goes live.
- Keep the capacity arithmetic in section 10 visible and end every
  estimate with what it changes in the design.
- Related: Event-Driven Architecture Design
  (`architecture/event-driven-architecture-design`), Event Sourcing and
  CQRS (`architecture/event-sourcing-cqrs`), Data Platform Architecture
  (`architecture/data-platform-architecture`).
-->

## Document control

<!-- Guidance: Identify the solution, the platforms it runs on and the owners. A change of key, partitioning scheme or delivery guarantee for a published stream is a new major version. -->

| Field | Value |
|---|---|
| Document ID | [STR-001] |
| Solution name | [Name] |
| Streaming platform | [Apache Kafka 4.x managed service / self-managed cluster / other], version [x.y] |
| Stream processor | [Apache Flink 2.x / Kafka Streams / other], version [x.y] |
| Schema registry | [Product and version] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@streaming-architect] |
| Approvers/reviewers | [@platform-owner], [@security-architect], [@data-platform-architect], [@sre-lead] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal / Confidential] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Business owner | [@name] | [Approve / Approve with conditions / Reject] | YYYY-MM-DD |
| Streaming platform owner | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| SRE | [@name] | [Decision] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Stock availability stream approved |
| 1.1 | YYYY-MM-DD | [@name] | *Example:* Added fraud scoring job and alert sink |

## Contents

<!-- Guidance: Keep this list aligned with the headings. -->

1. [Use cases and latency objectives](#1-use-cases-and-latency-objectives)
2. [Requirements and delivery guarantees](#2-requirements-and-delivery-guarantees)
3. [Sources and change data capture](#3-sources-and-change-data-capture)
4. [Event contracts and schema governance](#4-event-contracts-and-schema-governance)
5. [Topic design](#5-topic-design)
6. [Processing topology](#6-processing-topology)
7. [Time, windows and state](#7-time-windows-and-state)
8. [Exactly-once and idempotency](#8-exactly-once-and-idempotency)
9. [Sinks and serving](#9-sinks-and-serving)
10. [Capacity estimation](#10-capacity-estimation)
11. [Scaling and performance](#11-scaling-and-performance)
12. [Reliability and recovery](#12-reliability-and-recovery)
13. [Security and privacy](#13-security-and-privacy)
14. [Operations and lag monitoring](#14-operations-and-lag-monitoring)
15. [Testing](#15-testing)
16. [Trade-offs, roadmap and risks](#16-trade-offs-roadmap-and-risks)
17. [Glossary and references](#17-glossary-and-references)

## 1. Use cases and latency objectives

<!-- Guidance: State each use case with a measurable end-to-end latency objective: from the business event at the source to the result being usable. Questions: what is lost if the result is one minute late, one hour late? Pitfall: "real time" with no number, which leads to streaming where a 15-minute batch would do. -->

| ID | Use case | Consumer | Latency objective (p95) | Value of lower latency | Batch alternative rejected because |
|---|---|---|---|---|---|
| SUC-001 *(Example)* | Live stock availability per product and store | Web shop, store app | 30 seconds | Fewer orders for out-of-stock items | Hourly batch caused 2 % order cancellations |
| SUC-002 *(Example)* | Card-not-present fraud alert | Payment risk team, order service | 2 seconds | Order held before shipment | Fraud detected after dispatch |
| SUC-003 *(Example)* | Sales events to the lakehouse bronze layer | Data platform | 5 minutes | Intraday reporting | Hourly extracts load the POS database |
| [SUC-NNN] | [Use case] | [Consumer] | [Latency] | [Value] | [Reason] |

Record the objectives in the SLO document (`dev/slo-doc`) with the
measurement point at each end.

## 2. Requirements and delivery guarantees

<!-- Guidance: List functional and quality requirements, then state the delivery guarantee per output: at-most-once, at-least-once or effectively-once (exactly-once). Pitfall: demanding exactly-once everywhere, which costs latency, when idempotent consumers would do. -->

### 2.1 Requirements

| ID | Requirement | Type | Acceptance criterion |
|---|---|---|---|
| SRQ-001 *(Example)* | Compute available stock as on-hand minus reserved, per product and store | Functional | Matches the stock system snapshot within 0.5 % at daily reconciliation |
| SRQ-002 *(Example)* | Score each online payment with rules and a model | Functional | Score attached to 100 % of payment events |
| SRQ-003 *(Example)* | Handle peak of 50,000 events per second | Performance | Load test at 1.5 times peak without lag growth |
| SRQ-004 *(Example)* | No lost sales event after a broker or job failure | Reliability | Chaos test with zero missing events against source counts |
| SRQ-005 *(Example)* | Personal data encrypted in transit and access-controlled per topic | Security | Audit of ACLs and TLS configuration |
| [SRQ-NNN] | [Requirement] | [Type] | [Criterion] |

### 2.2 Delivery guarantee per output

| Output | Guarantee | How it is achieved | Reason |
|---|---|---|---|
| stock.availability topic *(Example)* | Effectively-once | Transactional sink committed on checkpoint; consumers read committed only | Downstream sums would drift with duplicates |
| Fraud alert to order service *(Example)* | At-least-once with idempotent receiver | Alert carries a deterministic identifier; receiver ignores repeats | 2-second objective rules out waiting for checkpoints |
| Bronze tables *(Example)* | Effectively-once | Sink commits files and table snapshot on checkpoint | Replays must not duplicate rows |
| [Output] | [Guarantee] | [Mechanism] | [Reason] |

## 3. Sources and change data capture

<!-- Guidance: Describe each producer, how events are created and the key they carry. Prefer business events emitted by the owning service; use CDC with the outbox pattern when the service cannot publish reliably. Pitfall: dual writes (database then broker) that lose events on failure. -->

| ID | Source | Method | Event types | Key | Rate (peak) | Owner |
|---|---|---|---|---|---|---|
| ESR-001 *(Example)* | POS gateway | Native producer, idempotent | SaleRecorded, ReturnRecorded | store_id | 30,000 per second | @retail-ops |
| ESR-002 *(Example)* | E-commerce order service | Transactional outbox read by CDC | OrderPlaced, OrderCancelled, PaymentAuthorised | order_id | 8,000 per second | @ecom-lead |
| ESR-003 *(Example)* | Warehouse system | Log-based CDC | StockAdjusted, StockReserved | product_id plus location_id | 12,000 per second | @supply-chain |
| [ESR-NNN] | [Source] | [Method] | [Types] | [Key] | [Rate] | [@owner] |

*Example* peak total: 30,000 + 8,000 + 12,000 = 50,000 events per
second. Section 10 sizes the platform from this figure. Snapshot,
ordering and schema change handling for CDC follow CDC Pipeline Design
(`system-design/design-cdc-pipeline`).

## 4. Event contracts and schema governance

<!-- Guidance: Define how events are described, versioned and validated. Questions: who approves a schema change, which compatibility mode applies, how do consumers find the contract? Pitfall: JSON without a schema, so any producer change breaks consumers at run time. -->

### 4.1 Contract rules

1. Describe each channel and message in an AsyncAPI 3.1.0 document
   (`asyncapi: 3.1.0`, with `info`, `servers`, `channels`, `operations`
   and `components`), stored with the producer's code and published to
   the catalogue. Detail it in Event API Specification
   (`architecture/event-api-specification`).
2. Register payload schemas (Avro, Protobuf or JSON Schema) in the schema
   registry; producers fail fast if the schema is not registered.
3. Use the envelope attributes of CloudEvents 1.0.2 (`id`, `source`,
   `type`, `specversion`, `time`) where events cross organisational
   boundaries, so routing does not depend on the payload.
4. Include an event identifier, the business event time and the producer
   version in every event.
5. Treat the topic, key, schema and guarantee of a published stream as a
   data contract (`ai-data/data-contract`).

### 4.2 Compatibility policy

| Stream class | Compatibility mode *(Example)* | Allowed change | Breaking change process |
|---|---|---|---|
| Internal to one team | Backward | Add fields with defaults, delete fields | New topic version, dual publish for [2] weeks |
| Shared across domains | Full transitive | Add optional fields with defaults only | New major contract version, consumer sign-off, [90] days dual publish |
| External partners | Full transitive plus AsyncAPI review | As above | Partner notice of [6] months |

Compatibility mode names follow common schema registry practice
(backward, forward, full, each optionally transitive, or none); check the
modes your registry supports.

## 5. Topic design

<!-- Guidance: Name topics consistently and set key, partitions, retention and cleanup per topic. The key decides ordering and parallelism. Pitfall: a low-cardinality key (country) that sends most traffic to one partition. -->

| Topic | Key | Partitions | Retention | Cleanup | Replication | Producers | Consumers |
|---|---|---|---|---|---|---|---|
| retail.sales.v1 *(Example)* | store_id | 24 | 7 days | Delete | 3, min in-sync 2 | POS gateway | Stock job, bronze sink |
| ecom.orders.v1 *(Example)* | order_id | 12 | 7 days | Delete | 3, min in-sync 2 | Outbox CDC | Fraud job, stock job, bronze sink |
| wms.stock-changes.v1 *(Example)* | product_id:location_id | 24 | 7 days | Delete | 3, min in-sync 2 | WMS CDC | Stock job, bronze sink |
| stock.availability.v1 *(Example)* | product_id:store_id | 24 | Latest value per key | Compact | 3, min in-sync 2 | Stock job | Web shop, store app |
| fraud.alerts.v1 *(Example)* | order_id | 6 | 30 days | Delete | 3, min in-sync 2 | Fraud job | Order service, risk team |
| [domain.entity.vN] | [Key] | [Count] | [Retention] | [Policy] | [RF] | [Producers] | [Consumers] |

Naming convention: `<domain>.<entity-or-event>.<major version>`. Where
an Event-Driven Architecture Design (`architecture/event-driven-architecture-design`)
exists, its convention (section 5.1, with environment and kind parts)
takes precedence; the short names here keep the examples readable. Keep a
dead-letter topic per consuming job (`<source-topic>.dlq.<job>`, the same
rule as the EDA design) with the original
payload, the error and the source offset.

*Example* skew check: store_id has 1,800 values over 24 partitions, about
75 stores per partition. The largest store produces [3] times the average
volume, so verify the hottest partition stays under the per-partition
limit measured in section 10.

## 6. Processing topology

<!-- Guidance: Show each job as a graph of sources, operators and sinks, with keys and parallelism. Questions: where are repartitions (shuffles), which operators hold state, which outputs are transactional? Pitfall: one large job for unrelated use cases, so a fraud rule change redeploys stock. -->

```mermaid
flowchart LR
  S1[retail.sales.v1] --> P1[Parse and validate]
  S2[ecom.orders.v1] --> P1
  S3[wms.stock-changes.v1] --> P1
  P1 -->|invalid| DLQ["retail.sales.v1.dlq.stock-job"]
  P1 --> K1[Key by product and store]
  K1 --> A1[Stateful availability: on-hand minus reserved]
  A1 --> O1[stock.availability.v1]
  A1 --> O2[Bronze table sink]
  S2 --> F1[Key by customer]
  F1 --> F2[Windowed features: 10-min count and amount]
  F2 --> F3[Rules and model score]
  F3 -->|score above threshold| O3[fraud.alerts.v1]
```

| Job | Operators with state | Parallelism *(Example)* | Deployment | Owner |
|---|---|---|---|---|
| stock-availability *(Example)* | Availability per product and store, deduplication by event id | 24 | Application mode on Kubernetes, one job per use case | @retail-data |
| fraud-scoring *(Example)* | Per-customer windows, model in memory | 12 | Application mode, separate cluster | @risk-data |
| [Job] | [Stateful operators] | [Parallelism] | [Deployment] | [@owner] |

## 7. Time, windows and state

<!-- Guidance: Decide event time versus processing time, watermark strategy, allowed lateness and window types; size state and set time-to-live. Pitfall: processing-time windows for business metrics, so a replay gives different answers. -->

### 7.1 Time and lateness

| Stream | Time used | Watermark strategy | Allowed lateness | Late events after lateness |
|---|---|---|---|---|
| retail.sales.v1 *(Example)* | Event time (till timestamp) | Bounded out-of-orderness of 30 seconds; idle source detection after 1 minute | 10 minutes | Side output to a correction topic, applied in the lakehouse |
| ecom.orders.v1 *(Example)* | Event time | Bounded out-of-orderness of 5 seconds | 1 minute | Scored without window features, flagged |
| [Stream] | [Time] | [Strategy] | [Lateness] | [Handling] |

A watermark at time t asserts that no further events at or before t are
expected. Watermarks advance per parallel source instance, and an
operator with several inputs follows the slowest; an idle partition can
therefore stall results, which is why idle source detection matters.

### 7.2 Windows

| ID | Window | Type | Size | Trigger | Use |
|---|---|---|---|---|---|
| WIN-001 *(Example)* | Customer payment features | Sliding | 10 minutes, every 1 minute | On watermark | Fraud score |
| WIN-002 *(Example)* | Store sales per minute | Tumbling | 1 minute | On watermark plus early firing every 10 seconds | Store dashboard |
| WIN-003 *(Example)* | Browsing session | Session | Gap of 30 minutes | On gap | Basket abandonment |
| [WIN-NNN] | [Window] | [Type] | [Size] | [Trigger] | [Use] |

### 7.3 State

| Operator | State content | Keys | Size per key | Total state | Time-to-live | Backend |
|---|---|---|---|---|---|---|
| Availability *(Example)* | On-hand, reserved, last event id | 2,000,000 product-store pairs | 2 KB | 4 GB | None (live entities) | Embedded key-value store on local disk, incremental checkpoints |
| Deduplication *(Example)* | Event ids seen | Events in last 15 minutes | 64 bytes | Section 10.4 | 15 minutes | As above |
| Customer windows *(Example)* | Payment amounts and counts | Active customers | 1 KB | [Size] | 24 hours | As above |
| [Operator] | [Content] | [Keys] | [Size] | [Total] | [TTL] | [Backend] |

Set a time-to-live on every state that is not a live entity; state that
grows without bound is the most common cause of slow checkpoints and
long recovery.

## 8. Exactly-once and idempotency

<!-- Guidance: Explain how each output avoids loss and duplicates through failures and restarts. Exactly-once applies to state and to sinks that take part in the checkpoint; everything else needs idempotency. Pitfall: assuming the broker's exactly-once covers writes to an external database. -->

| Layer | Mechanism *(Example)* | Configuration to check |
|---|---|---|
| Producers | Idempotent producer with acknowledgement from all in-sync replicas | acks set to all, idempotence enabled, retries enabled |
| Processor state | Consistent checkpoints of all operator state and source offsets | Checkpoint interval [60 s], minimum pause, timeout, retained checkpoints |
| Broker sink | Transactional writes committed when a checkpoint completes | Unique transactional id prefix per job; transaction timeout longer than the longest checkpoint plus restart time |
| Downstream consumers | Read committed data only | Consumer isolation level read committed |
| External database sink | Idempotent upsert keyed by business key plus event version | Upsert ignores versions older than stored |
| Alerts | Deterministic alert id derived from order id and rule id | Receiver stores processed ids for [7] days |

Consequence of the transactional sink: results become visible to
read-committed consumers only after the checkpoint completes. With a
60-second interval the stock topic can lag by up to about a minute plus
commit time, which breaks the 30-second objective in SUC-001. *Example*
choice: checkpoint the stock job every 10 seconds (state is about 6.9 GB, section 10.4, and
checkpoints are incremental) and keep the fraud alert path out of the
transaction, as stated in section 2.2. Record the decision in the
Architecture Decision Log (`architecture/architecture-decision-log`).

## 9. Sinks and serving

<!-- Guidance: List every output and how consumers read it. Pitfall: a sink that cannot keep up and silently backpressures the whole job. -->

| ID | Sink | Data | Write pattern | Guarantee | Consumer |
|---|---|---|---|---|---|
| SNK-001 *(Example)* | Compacted topic stock.availability.v1 | Latest availability per key | Upsert by key | Effectively-once | Web shop cache, store app |
| SNK-002 *(Example)* | Key-value cache | Availability for page rendering | Upsert from SNK-001 by a consumer service | At-least-once, idempotent | Web shop |
| SNK-003 *(Example)* | Lakehouse bronze table | All sales, order and stock events | Append, commit on checkpoint, compaction daily | Effectively-once | Data platform |
| SNK-004 *(Example)* | fraud.alerts.v1 and order service API | Alerts | Publish then call with retries | At-least-once, idempotent | Order service |
| [SNK-NNN] | [Sink] | [Data] | [Pattern] | [Guarantee] | [Consumer] |

Streaming sinks into the lakehouse produce many small files; schedule
compaction as described in Data Platform Architecture
(`architecture/data-platform-architecture`).

## 10. Capacity estimation

<!-- Guidance: Derive throughput, storage, partitions and state from the source rates. Show each calculation, state the unit convention, and end with the consequence for the design. Pitfall: sizing for the average and failing on the first promotion day. -->

Units in this section are decimal: 1 KB = 1,000 bytes, 1 MB = 10^6
bytes, 1 TB = 10^12 bytes. All figures are *Example* values.

### 10.1 Throughput

| Step | Calculation | Result |
|---|---|---|
| Peak ingress | 50,000 events per second x 1 KB average | 50 MB/s |
| Replicated write at replication factor 3 | 50 MB/s x 3 | 150 MB/s across brokers |
| Consumer egress (sales read by two groups, orders by three, stock changes by two; section 5) | 30 MB/s x 2 + 8 MB/s x 3 + 12 MB/s x 2 | 108 MB/s |

Consequence: brokers must sustain 150 MB/s of disk writes plus 100 MB/s
of replication traffic between brokers (two follower copies of 50 MB/s)
and about 108 MB/s of consumer reads, mostly served from page cache;
size network and disks with [50 %] headroom.

### 10.2 Retention storage

| Step | Calculation | Result |
|---|---|---|
| Average daily volume | 20,000 events per second x 1 KB x 86,400 s | 1.728 TB per day |
| Seven days logical | 1.728 TB x 7 | 12.096 TB |
| With replication factor 3 | 12.096 TB x 3 | 36.288 TB |

Consequence: about 36 TB of broker storage for 7 days. If the platform
offers tiered storage, keep [1] day on local disk and the rest in object
storage, which cuts local disk to about 1.728 x 3 = 5.2 TB.

### 10.3 Partitions

| Step | Calculation | Result |
|---|---|---|
| Per-partition throughput measured in a load test | Given | 5 MB/s |
| Peak on retail.sales.v1 (POS stream only, ESR-001) | 30,000 events per second x 1 KB | 30 MB/s |
| Minimum partitions for peak | 30 MB/s / 5 MB/s | 6 |
| Minimum partitions for twice today's peak | 60 MB/s / 5 MB/s | 12 |
| Chosen for retail.sales.v1 | Three-year growth and stock job parallelism; divides evenly by 8, 12 and 24 | 24 |

Consequence: 24 partitions cap the parallelism of any consumer group at
24. Increasing partitions later changes key-to-partition mapping and
breaks per-key ordering for in-flight data, so choose the count for the
three-year growth now.

### 10.4 State

| Step | Calculation | Result |
|---|---|---|
| Availability state | 2,000,000 keys x 2 KB | 4 GB |
| State per subtask at parallelism 24 | 4 GB / 24 | about 167 MB |
| Deduplication state | 50,000 events per second x 900 s x 64 bytes | 2.88 GB |
| Total stock job state | 4 GB + 2.88 GB | about 6.9 GB |

Consequence: about 290 MB of state per subtask (6.9 GB / 24) restores in
seconds from local disk and in under a minute from object storage at
typical throughput; 10-second incremental checkpoints are feasible. If
deduplication by event id were held for 24 hours instead of 15 minutes,
it would grow 96 times to about 276 GB, so keep the window short and rely
on idempotent producers for older duplicates.

## 11. Scaling and performance

<!-- Guidance: Describe how each component scales and what limits it. Questions: what is the first bottleneck at twice the load, how is backpressure detected? Pitfall: scaling the processor beyond the partition count, which leaves subtasks idle. -->

| Component | Scaling method | Limit | Trigger *(Example)* |
|---|---|---|---|
| Brokers | Add brokers and reassign partitions | Disk and network per broker | Disk above 70 % or network above 60 % |
| Stream jobs | Rescale parallelism from a savepoint, or autoscaler | Partition count of the input topic | Consumer lag growing for 5 minutes |
| Sinks | Scale the receiving store; batch writes | Write capacity of the store | Backpressure on sink operator above 50 % |
| Schema registry | Replicas behind a load balancer | Rarely the bottleneck | Request latency above 50 ms |

Performance targets: p95 end-to-end latency per SUC objective, measured
by comparing event time with sink write time on sampled events. Plan load
tests in the Performance Test Plan (`quality/performance-test-plan`).

## 12. Reliability and recovery

<!-- Guidance: State failure modes and what happens in each, plus recovery objectives and cross-region strategy. Pitfall: savepoints never tested, so an upgrade loses state. -->

| Failure *(Example)* | Effect | Detection | Recovery |
|---|---|---|---|
| One broker fails | Leaders move to other replicas; producers retry | Under-replicated partitions alert | Automatic; replace broker |
| Job task manager fails | Job restarts from last checkpoint; output replays to last commit | Restart counter, lag | Automatic within [60 s] |
| Poison message | Parsing fails | Dead-letter topic rate | Message routed to DLQ, job continues |
| Downstream cache unavailable | Consumer service lags | Lag on its consumer group | Retry with backoff, cache rebuilt from compacted topic |
| Region outage | Platform unavailable in the region | Health checks | Fail over to the standby region per resilience tier |

Recovery objectives: RPO [1 minute] and RTO [30 minutes] for the stock
stream (tier per Resilience Tiering and HA/DR Design,
`architecture/resilience-tiering-ha-dr-design`). Cross-region replication
copies topics asynchronously; consumer offsets must be translated, and
stateful jobs restart from the latest replicated savepoint or rebuild
state by replaying compacted topics. Take a savepoint before every job
upgrade and test restore quarterly.

## 13. Security and privacy

<!-- Guidance: Cover authentication, authorisation per topic, encryption, personal data in events and retention. Pitfall: personal data in keys, which then appears in logs and partitioner metrics. -->

| Control | Design *(Example)* |
|---|---|
| Authentication | Mutual TLS or OAuth-based authentication for every client; one identity per producer and job |
| Authorisation | ACLs per topic and consumer group; producers write only their topics |
| Encryption | TLS in transit; encryption at rest on brokers and tiered storage |
| Personal data | No personal data in keys; customer identifiers pseudonymised in sales events; card data never on the stream (tokens only) |
| Retention and erasure | Short retention (7 days) on raw topics; compacted topics with personal data support deletion by key (tombstone) |
| Audit | Admin operations and ACL changes logged to the security log store |
| Threat analysis | Producer impersonation, topic flooding, data exfiltration through consumer groups; recorded in `security/threat-model` |

Assess personal data processing in the DPIA (`compliance/dpia`) when
events feed profiling such as fraud scoring.

## 14. Operations and lag monitoring

<!-- Guidance: List the signals, alerts and runbooks that keep the solution healthy. Consumer lag in time, not only in messages, is the main health signal. Pitfall: alerting on lag in messages, which means nothing when traffic varies tenfold. -->

| Signal | Threshold *(Example)* | Action |
|---|---|---|
| Consumer lag in seconds (event time of last processed record versus now) | Above 2 times the latency objective for 5 minutes | Page on-call; check backpressure and sink health |
| Checkpoint duration | Above 50 % of the interval | Investigate state growth and storage |
| Checkpoint failures | 3 in a row | Page on-call |
| Job restarts | More than 3 per hour | Page on-call |
| Dead-letter rate | Above 0.1 % of input | Notify producer owner |
| Under-replicated partitions | Any for 5 minutes | Platform on-call |
| Schema registration rejected | Any | Notify producer owner |

Write signal definitions and dashboards in the Observability
Specification (`ops/observability-spec`) and runbooks for lag, failed
checkpoints, DLQ replay and savepoint restore in `dev/runbook`.

## 15. Testing

<!-- Guidance: Test logic, time handling, failure behaviour and performance. Pitfall: unit tests with in-order events only, which never exercise watermarks and late data. -->

| ID | Test | Method | Pass criterion |
|---|---|---|---|
| STT-001 *(Example)* | Operator logic | Unit tests with a test harness that controls time | Expected outputs for in-order, out-of-order and late events |
| STT-002 *(Example)* | Contract compatibility | Schema check in CI against the registry | No incompatible change merged |
| STT-003 *(Example)* | Exactly-once under failure | Kill task managers and brokers during a replay of known data | Output equals expected with no duplicates or gaps |
| STT-004 *(Example)* | Load | Replay of a peak day at 1.5 times rate | Lag stable, latency objectives met |
| STT-005 *(Example)* | Reconciliation | Daily compare of stream totals with source system totals | Within 0.5 % (SRQ-001) |
| STT-006 *(Example)* | Upgrade | Savepoint, upgrade, restore in staging | State restored, no lost events |
| [STT-NNN] | [Test] | [Method] | [Criterion] |

## 16. Trade-offs, roadmap and risks

<!-- Guidance: Record the main design trade-offs, the delivery plan and the risks. Pitfall: no exit plan from a managed streaming service. -->

### 16.1 Trade-offs

| Decision | Chosen *(Example)* | Alternative | Why chosen | Revisit when |
|---|---|---|---|---|
| Processor | Flink-style engine with event time and managed state | Broker-native stream library | Several input topics, large state, event-time windows | Use cases become simple per-topic transforms |
| Stock output | Compacted topic plus cache | Direct writes to the cache | Replayable, decouples consumers | Cache supports change streams natively |
| Fraud path | At-least-once with idempotent receiver | Transactional sink | 2-second objective | Checkpoint interval can drop below 1 second |
| Hosting | Managed streaming service | Self-managed cluster | Operations effort | Cost at ten times volume exceeds a dedicated team |

### 16.2 Roadmap and risks

| Increment | Scope | Exit criterion |
|---|---|---|
| S1 *(Example)* | Sales and stock topics, bronze sink | Reconciliation within 0.5 % for 4 weeks |
| S2 *(Example)* | Stock availability job and cache | SUC-001 objective met in production |
| S3 *(Example)* | Fraud scoring | False positive rate agreed with risk team |

| ID | Risk | Likelihood | Impact | Response | Owner |
|---|---|---|---|---|---|
| RSK-001 *(Example)* | Hot partition from a flagship store | Medium | Medium | Key salting for that store or a finer key | @streaming-architect |
| RSK-002 *(Example)* | Schema change breaks consumers | Medium | High | Registry enforcement, STT-002 in CI | @platform-owner |
| RSK-003 *(Example)* | State growth slows checkpoints | Medium | High | TTL on all state, checkpoint alert | @sre-lead |
| [RSK-NNN] | [Risk] | [L / M / H] | [L / M / H] | [Response] | [@owner] |

## 17. Glossary and references

<!-- Guidance: Define streaming terms for non-specialists and record versions checked, with dates. -->

| Term | Meaning |
|---|---|
| Event time | Time at which the event happened at the source |
| Watermark | Marker asserting that no earlier events are expected, which lets windows close |
| Allowed lateness | Extra time during which late events still update a closed window |
| Checkpoint | Consistent snapshot of operator state and input positions used for recovery |
| Savepoint | User-triggered snapshot used for upgrades, rescaling and migration |
| Compacted topic | Topic that keeps at least the latest record per key |
| Consumer lag | How far a consumer is behind the end of a partition, in records or time |
| Effectively-once | Each input affects the results exactly once despite retries and restarts |
| Outbox pattern | Writing an event to a database table in the same transaction as the state change, then publishing it |

| Reference | Version or date checked | Use |
|---|---|---|
| AsyncAPI Specification, asyncapi.com | 3.1.0, released 2026-01-31, checked 2026-10-23 | Event contracts (section 4) |
| CloudEvents, cloudevents.io (CNCF) | 1.0.2 | Envelope attributes |
| Apache Kafka documentation and upgrade notes, kafka.apache.org | 4.3 line (4.3.1 released 2026-06-25); KRaft only since 4.0; share groups production-ready since 4.2; checked 2026-10-23 | Platform concepts |
| Apache Flink documentation, flink.apache.org | 2.3.0 (2026-06-25); 1.20 is the LTS line; checked 2026-10-23 | Time, watermarks, windows, exactly-once Kafka sink |
| Schema registry documentation (for example Confluent Schema Registry, docs.confluent.io) | Checked 2026-10-23 | Compatibility mode names |
| [Managed service documentation] | [Version] | [Limits and quotas] |

## Standard and licence note

Based on the AsyncAPI Specification 3.1.0 (Linux Foundation, Apache
License 2.0) for event contracts and CloudEvents 1.0.2 (CNCF) for event
envelopes, with concepts described from the public Apache Kafka and
Apache Flink documentation (Apache License 2.0). The text and the worked
design are original and no documentation text is reproduced. This
template is original text written for this collection. Check the current
official versions of each specification and product before use. Apache
Kafka and Apache Flink are trademarks of the Apache Software Foundation,
and Confluent is a trademark of its owner; this template is not
affiliated with or endorsed by them. Template licence: MIT.