# Technical Design Document: [Feature or system name]

> In-depth technical design document for a significant feature, service or
> migration, in the style of engineering design docs used at large
> software companies: context, goals and non-goals, detailed design,
> alternatives, cross-cutting concerns, testing, rollout and cost. For a
> short proposal, use the lean Design Doc template instead. Replace every
> `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- Write sections 1 to 3 first and circulate them before the detailed
  design. Most expensive mistakes are wrong goals, not wrong code.
- Section 5 (alternatives) is mandatory. Reviewers trust a design that
  shows the options it rejected and why.
- Keep the document about trade-offs. Implementation detail that will be
  obvious from the code can be left out.
- Close every open question in section 11 before approval, or record who
  decides and by when.
-->

## Document control

<!-- Guidance: Who wrote it, who must review it, and where it stands. Reviewers listed here are expected to approve or comment. -->

| Field | Value |
|---|---|
| Document ID | [TDD-YYYY-NNN] |
| Title | [Feature or system name] |
| Author(s) | [@author] |
| Reviewers | [@reviewer1, @reviewer2, @security-reviewer] |
| Status | [Draft / In review / Approved / Implemented / Abandoned] |
| Target release | [vX.Y / quarter] |
| Tracking issue | [link] |
| Classification | [Internal / Confidential] |
| Created / last updated | YYYY-MM-DD / YYYY-MM-DD |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Approved with comments / Rejected] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| Security reviewer | [@name] | [Decision] | YYYY-MM-DD |
| Product manager | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Summarise substantive changes after review rounds so reviewers can see what moved. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | Context, goals and overview for early feedback |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Detailed design and alternatives after first review |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved; rollout plan finalised |

## Contents

1. [Context and scope](#1-context-and-scope)
2. [Goals and non-goals](#2-goals-and-non-goals)
3. [Overview of the design](#3-overview-of-the-design)
4. [Detailed design](#4-detailed-design)
5. [Alternatives considered](#5-alternatives-considered)
6. [Cross-cutting concerns](#6-cross-cutting-concerns)
7. [Testing strategy](#7-testing-strategy)
8. [Rollout and migration](#8-rollout-and-migration)
9. [Operations and support](#9-operations-and-support)
10. [Cost](#10-cost)
11. [Open questions](#11-open-questions)
12. [References and glossary](#12-references-and-glossary)

## 1. Context and scope

<!-- Guidance: Give a reader with no background enough context to judge the design: the system today, the problem, and why now. Facts, not proposals. Keep it short. -->

### 1.1 Background

[Describe the current system and how it behaves today. Example: "Search
on the catalogue uses SQL LIKE queries against the product table. With
1.2 million products, p95 search latency is 2.8 s and full-text relevance
is poor: 31% of searches end without a click."]

### 1.2 Problem statement

[The specific problem this design solves, with evidence: metrics,
incidents, customer feedback, upcoming load.]

### 1.3 Scope

| In scope | Out of scope |
|---|---|
| *Example:* Product search API and indexing pipeline | Search UI redesign (separate project) |
| *Example:* Migration of existing search traffic | Personalised ranking |
| [Item] | [Item] |

## 2. Goals and non-goals

<!-- Guidance: Goals are measurable outcomes. Non-goals are things a reasonable reader might expect but that this design deliberately does not do. A good non-goal prevents a review argument. -->

### 2.1 Goals

| ID | Goal | Measure | Target |
|---|---|---|---|
| G-01 | *Example:* Fast search | p95 latency of `GET /search` | Under 200 ms at 300 requests/s |
| G-02 | *Example:* Relevant results | Searches ending without click | Below 15% |
| G-03 | *Example:* Fresh index | Delay from product update to searchable | Under 60 s at p99 |
| [G-nn] | [Goal] | [Metric] | [Target] |

### 2.2 Non-goals

- [Example: Supporting search across marketplace partner catalogues.]
- [Example: Replacing the product database as the source of truth.]
- [Example: Multi-language stemming beyond English and German in this phase.]

## 3. Overview of the design

<!-- Guidance: One or two paragraphs and one diagram a reviewer can understand in two minutes. Detail goes in section 4. -->

[Summary. Example: "We introduce a search index (OpenSearch) populated by
change events from the product database through an indexer service. The
search API queries the index and falls back to the existing SQL search if
the index is unavailable."]

```mermaid
flowchart LR
  client[Web and mobile clients] --> api[Search API]
  api --> idx[(Search index)]
  api -. fallback .-> db[(Product database)]
  db -->|change events| cdc[Change data capture]
  cdc --> indexer[Indexer service]
  indexer --> idx
```

## 4. Detailed design

<!-- Guidance: The core of the document. Cover interfaces, data, component changes and behaviour, including failure paths. Prefer tables and diagrams to prose. -->

### 4.1 APIs

| Method and path | Request | Response | Errors | Notes |
|---|---|---|---|---|
| *Example:* `GET /v2/search` | `q`, `filters`, `page`, `pageSize` (max 100) | `results[]`, `facets`, `total`, `nextCursor` | 400 invalid filter, 429, 503 | Cursor-based paging; v1 kept for 6 months |
| [Endpoint] | [Request] | [Response] | [Errors] | [Notes] |

```json
{
  "q": "trail running shoes",
  "filters": {"brand": ["Acme"], "size": ["42"]},
  "pageSize": 24
}
```

Compatibility: [how existing clients are affected; versioning approach;
deprecation timeline].

### 4.2 Data storage

| Store | Data | Schema / mapping | Size | Retention | Owner |
|---|---|---|---|---|---|
| *Example:* Search index `products_v1` | Denormalised product documents | Text fields with English and German analysers; keyword fields for facets | 1.2 M docs, about 6 GB | Rebuilt on mapping change | Search team |
| *Example:* Product database | Source of truth, unchanged | Existing schema | 40 GB | Existing policy | Catalogue team |
| [Store] | [Data] | [Schema] | [Size] | [Retention] | [Team] |

Consistency model: [for example "the index is eventually consistent with
the database; G-03 bounds the delay; reads that require exact stock levels
still go to the database"].

### 4.3 Code and module changes

| Component | Change | Owner | Size estimate |
|---|---|---|---|
| *Example:* `search-api` (new) | Query parsing, index client, fallback | Search team | 3 weeks |
| *Example:* `indexer` (new) | Consume change events, transform, bulk index | Search team | 2 weeks |
| *Example:* `catalogue-service` | Emit change events via outbox | Catalogue team | 1 week |
| [Component] | [Change] | [Team] | [Estimate] |

### 4.4 Key flows

The indexing flow and its failure handling:

```mermaid
sequenceDiagram
  participant Cat as Catalogue service
  participant DB as Product database
  participant CDC as Change data capture
  participant Ix as Indexer
  participant OS as Search index
  Cat->>DB: Update product
  DB-->>CDC: Row change
  CDC->>Ix: ProductChanged(id, version)
  Ix->>DB: Read full product
  Ix->>OS: Upsert document if version is newer
  OS-->>Ix: OK
  Note over Ix,OS: On failure, retry 5 times with backoff, then dead-letter queue and alert
```

### 4.5 Failure modes and degradation

| Failure | Impact | Detection | Behaviour |
|---|---|---|---|
| *Example:* Index cluster unavailable | Search fails | Health check, error rate | Search API falls back to SQL search with reduced features; banner not shown to users |
| *Example:* Indexer lag | Stale results | Lag metric above 60 s | Alert; autoscale indexer |
| *Example:* Bad mapping deployed | Wrong results | Relevance canary queries | Switch alias back to previous index |
| [Failure] | [Impact] | [Signal] | [Behaviour] |

### 4.6 Capacity

Index throughput needed during a full rebuild:

$$
t_{rebuild} = \frac{N_{docs}}{r_{bulk}} = \frac{1\,200\,000}{2\,000\ \text{docs/s}} = 600\ \text{s}
$$

[State load assumptions, node sizing and headroom. Example: "3 data
nodes with 8 vCPU and 32 GB each handle 300 queries/s at 35% CPU in the
load test; we provision for 2x peak."]

## 5. Alternatives considered

<!-- Guidance: Describe each serious alternative fairly, with the criteria used to compare. "Do nothing" is always an option. -->

| Criterion | Weight | A: Dedicated search index (chosen) | B: PostgreSQL full-text search | C: Hosted search SaaS | D: Do nothing |
|---|---|---|---|---|---|
| Meets latency goal G-01 | High | Yes | Partly (p95 about 600 ms in test) | Yes | No |
| Relevance tuning | High | Full control | Limited | Good | None |
| Operational effort | Medium | New cluster to run | None new | Low | None |
| Data leaves our infrastructure | High | No | No | Yes | No |
| Cost per month | Medium | [USD 2,100] | [USD 300] | [USD 4,500] | 0 |

Reasons for the choice: [Example: "B fails the latency goal at our data
volume; C is excluded by the data residency policy; A meets all goals at
acceptable operational cost."]

## 6. Cross-cutting concerns

<!-- Guidance: Address each concern explicitly, even if the answer is "no change". Reviewers from each area will look for their heading. -->

### 6.1 Security

[Authentication and authorisation of new endpoints, input validation and
query injection risk, secrets handling, network exposure, dependency
review. Example: "The index is reachable only from the search API and
indexer subnets; queries are built with the client library, never by
string concatenation."]

### 6.2 Privacy

[Personal data processed or stored, lawful basis, retention, deletion
propagation. Example: "No personal data is indexed. Seller names are
public business data."]

### 6.3 Observability

| Signal | Metric / log / trace | Alert threshold | Dashboard |
|---|---|---|---|
| *Example:* Search latency | `search_request_duration_seconds` histogram | p95 above 300 ms for 10 min | Search overview |
| *Example:* Indexer lag | `indexer_lag_seconds` | Above 120 s for 5 min | Indexing |
| [Signal] | [Instrument] | [Threshold] | [Dashboard] |

### 6.4 Accessibility

[Impact on user-facing surfaces. Example: "API only. Client teams keep
announcing result counts to screen readers when results change."]

### 6.5 Internationalisation

[Languages, analysers, collation, number and date formats. Example:
"English and German analysers in phase 1; queries in other languages
fall back to the standard analyser."]

### 6.6 Reliability and data integrity

[Idempotency, ordering, versioning of documents, reconciliation job that
compares database and index counts daily.]

## 7. Testing strategy

<!-- Guidance: Show how you will know the design works before and after launch, including relevance or correctness checks that unit tests cannot cover. -->

| Level | What is tested | Tooling | Exit criterion |
|---|---|---|---|
| Unit | Query parsing, document mapping | [Test framework] | 90% branch coverage on new code |
| Integration | Indexer against a real index container | [Testcontainers] | All flows in 4.4 pass |
| Contract | `GET /v2/search` against client expectations | [Contract testing tool] | No breaking changes |
| Performance | 300 requests/s for 1 hour | [Load tool] | G-01 met |
| Relevance | 500 judged queries | Offline evaluation | NDCG at 10 at least 0.05 above current |
| Resilience | Kill index node, stop indexer | Chaos test | Fallback and recovery as in 4.5 |

## 8. Rollout and migration

<!-- Guidance: How the change reaches production safely, how you will know it is working, and how to undo it. -->

### 8.1 Rollout plan

| Phase | Audience | Duration | Success criteria | Rollback |
|---|---|---|---|---|
| 1. Shadow | 100% traffic mirrored, results discarded | 1 week | Error rate under 0.1%, latency within goal | Disable mirroring |
| 2. Canary | 5% of users via feature flag | 1 week | Click-through not lower than control | Flag off |
| 3. Ramp | 25%, 50%, 100% | 2 weeks | Same as phase 2 | Flag off |
| 4. Clean-up | Remove old path and v1 API | After 6 months | No v1 traffic for 30 days | n/a |

### 8.2 Data migration

[Initial backfill procedure, duration, verification (counts and sampled
document comparison), and how changes during backfill are captured.]

| Step | Action | Verification | Reversible |
|---|---|---|---|
| 1 | *Example:* Start change data capture and buffer events | Connector lag visible on dashboard | Yes |
| 2 | *Example:* Bulk-load all products into `products_v1` | Document count equals database count within 0.01% | Yes, drop index |
| 3 | *Example:* Replay buffered events | Lag returns to under 60 s | Yes |
| 4 | *Example:* Point alias `products` to `products_v1` | Canary queries return expected top results | Yes, switch alias |

### 8.3 Launch checklist

| Item | Owner | Done |
|---|---|---|
| Security review completed and findings closed | [@security] | [Yes / No] |
| Privacy review (if personal data is affected) | [@privacy] | [Yes / No / Not applicable] |
| Dashboards and alerts from 6.3 live and tested | [@owner] | [Yes / No] |
| Runbooks written and rehearsed by on-call | [@owner] | [Yes / No] |
| Load test at 2x expected peak passed | [@owner] | [Yes / No] |
| Rollback rehearsed in staging | [@owner] | [Yes / No] |
| Client teams and support informed of the change and timeline | [@pm] | [Yes / No] |

## 9. Operations and support

<!-- Guidance: Who runs the new parts, what they need, and what changes for on-call. -->

| Item | Detail |
|---|---|
| Owning team and on-call | [Search team, primary rotation] |
| Runbooks | [Links: index recovery, alias switch, reindex] |
| Service level objective | [99.9% of searches succeed in under 300 ms over 28 days] |
| Dependencies added | [Search cluster, change data capture connector] |
| Support impact | [None expected / FAQ for support agents] |

## 10. Cost

<!-- Guidance: Show recurring and one-off costs with the assumptions. Include people time for operations. -->

| Item | One-off | Monthly | Assumption |
|---|---|---|---|
| *Example:* Search cluster (3 nodes) | 0 | USD 1,800 | On-demand pricing, region [x] |
| *Example:* Change data capture | 0 | USD 300 | Managed connector |
| *Example:* Engineering | 8 engineer-weeks | 0.1 FTE operations | Section 4.3 estimates |
| [Item] | [Amount] | [Amount] | [Assumption] |

## 11. Open questions

<!-- Guidance: Each question has an owner and a date. Move answered questions into the relevant section and note the answer here. -->

| ID | Question | Owner | Due | Answer |
|---|---|---|---|---|
| OQ-01 | *Example:* Do we need synonyms per market at launch? | @pm | YYYY-MM-DD | [Open / answer] |
| OQ-02 | *Example:* Can the catalogue team ship outbox events before sprint [n]? | @catalogue-lead | YYYY-MM-DD | [Open / answer] |
| [OQ-nn] | [Question] | [@owner] | YYYY-MM-DD | [Answer] |

## 12. References and glossary

<!-- Guidance: Link prior designs, incidents, benchmarks and terms a newcomer would not know. -->

| Reference | Link |
|---|---|
| [Previous design or ADR] | [link] |
| [Load test results] | [link] |
| [Incident report motivating this work] | [link] |

| Term | Meaning |
|---|---|
| *Example:* Alias | A stable index name pointing to a versioned physical index, switched atomically |
| *Example:* NDCG | Normalised discounted cumulative gain, a ranking quality measure |
| [Term] | [Meaning] |

## Standard and licence note

Based on common industry practice for engineering design documents, as
used by large software organisations: context, goals and non-goals,
design, alternatives, cross-cutting concerns and rollout. Template
licence: MIT.