# System Design: [LLM gateway name]

> Reference design for an LLM gateway and model router: one internal API in
> front of several model providers and self-hosted models. It covers the
> unified API, provider adapters, keys and quotas per tenant, routing and
> fallback, exact and semantic caching, PII redaction, guardrail hooks,
> token streaming, cost metering, telemetry with the OpenTelemetry semantic
> conventions for generative AI (Development status at the time of writing)
> and the audit trail. Model serving itself is covered by the LLM Inference
> Serving and Vector Search Platform Design Template
> (`system-design/design-llm-inference-serving`). It follows the 12-section
> structure of the System Design Document Template
> (`system-design/system-design-document`) and is pre-filled with a worked
> design to adapt. Replace every `[bracketed]` value and every row marked
> *Example*; all numbers are illustrative, not benchmarks or vendor prices.

<!--
Guidance: How to use this template
- Start from who calls the gateway and why: count tenants, use cases,
  providers and the data classes that flow through prompts (2.1, 3.1).
- Decide early whether the gateway is on the critical path for every LLM call
  in the company. If it is, its availability target must be higher than any
  single provider's, which drives multi-region and fallback design (7.3).
- Keep the gateway thin: authentication, quotas, routing, redaction hooks,
  metering and telemetry. Business prompts, retrieval and agent logic belong
  to the calling applications.
- Re-check provider limits, prices and the OpenTelemetry GenAI conventions
  before approval; all three change often.
-->

## Document control

<!-- Guidance: A design is a proposal until approved. Record who decided, and which providers and data classes the approval covers. -->

| Field | Value |
|---|---|
| Document ID | [SDD-LGW-001] |
| System / feature | [LLM gateway and model router] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Approvers / reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Providers in scope | [External providers, self-hosted pools, regions] |
| Data classes allowed in prompts | [Public / Internal / Confidential / Personal data with redaction] |
| Telemetry convention | [OpenTelemetry GenAI semantic conventions, version pinned: x.y] |
| Related | [API gateway design], [LLM observability spec], [Guardrails spec], [Threat model] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead, AI platform | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| Security and privacy | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| FinOps / budget owner | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Record design changes, not wording changes. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added semantic cache, off by default, opt-in per route |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Moved PII redaction before the cache lookup |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page a busy reviewer can read alone: the problem, the proposed solution in a paragraph and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Forty product teams call three external model providers and two
self-hosted model pools directly. Each team stores its own provider keys,
several of them in application configuration. Nobody can answer how much the
company spent on model calls last week, per team, until the invoices arrive.
Two incidents in the last quarter came from customer personal data sent to a
provider that is not approved for it. When one provider had a regional
outage, every team improvised its own fallback, and two of them silently
switched to a cheaper model with lower answer quality.]

### 1.2 Proposed solution

[*Example:* A single LLM gateway exposes one OpenAI-compatible chat and
embedding API to internal callers. Callers authenticate with a gateway key
bound to a tenant and a use case; the gateway holds all provider credentials.
For each request it checks quotas in tokens and requests, applies the tenant's
data policy (redaction, allowed providers, allowed regions), looks up the
exact cache, picks a route from an alias such as `chat-large` to a concrete
provider and model, streams the answer back, and emits one usage record and
one trace per call. Guardrail hooks run before and after the model call.
Fallback to another provider or model happens only along routes the tenant
has approved, and is visible in the response and in metering.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Unified API shape? | OpenAI-compatible chat and embeddings, with gateway extensions in headers | 4 |
| Q-02 | *Example:* Where does redaction run? | In the gateway, before cache lookup and before any external provider | 7.4 |
| Q-03 | *Example:* Semantic cache? | Off by default; opt-in per route with a similarity threshold and tenant scoping | 7.5 |
| Q-04 | *Example:* Fallback across providers? | Pre-approved fallback chains per alias, never across data-policy boundaries | 7.3 |
| Q-05 | *Example:* Quota unit? | Tokens per minute and a monthly budget in currency, with requests per minute as a secondary guard | 7.2 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: Use this order when presenting the design in an interview or a time-boxed review. Pick one deep dive, not all of them. -->

| Minutes | Topic | What to show |
|---|---|---|
| 0 to 5 | Scope | Callers, providers, the four jobs of the gateway (identity, policy, routing, metering); state that model serving is out of scope |
| 5 to 10 | Requirements | Core list from 2.1, two or three numbers from 2.2, the below-the-line list from 2.3 |
| 10 to 15 | Estimates | Concurrent streams, peak tokens per second and daily spend (3.2 to 3.5), each with its consequence |
| 15 to 22 | API and data | One request example (4.2), the error contract and the usage record (5.1) |
| 22 to 30 | High-level design | The diagram in 6, the request sequence in 7.1 |
| 30 to 40 | One deep dive | Routing and fallback (7.3) for reliability questions, or redaction (7.4) for privacy questions, or quotas (7.2) for multi-tenancy questions |
| 40 to 45 | Wrap-up | Failure modes (9.1), the 10x row of the trade-off table (10) |

## 2. Requirements

<!-- Guidance: Split what the design must do (core) from what is explicitly out of scope (below the line). Give every non-functional requirement a number and a measurement. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* One chat completion endpoint with streaming, and one embedding endpoint, compatible with the OpenAI request shape | Must |
| FR-02 | *Example:* Provider adapters translate the unified request to each provider's API and back, including streaming events, tool calls and errors | Must |
| FR-03 | *Example:* Gateway keys per tenant and use case; provider credentials held only by the gateway | Must |
| FR-04 | *Example:* Quotas per tenant and alias in tokens per minute, requests per minute and a monthly currency budget | Must |
| FR-05 | *Example:* Model aliases mapped to routes with weights, health checks and approved fallback chains | Must |
| FR-06 | *Example:* PII detection and redaction before external providers, with restoration in the response where the policy allows | Must |
| FR-07 | *Example:* Guardrail hooks before and after the model call, synchronous or asynchronous per policy | Must |
| FR-08 | *Example:* Usage metering per request: tenant, alias, route, input, output and cached tokens, cost | Must |
| FR-09 | *Example:* Exact-match response cache for deterministic requests | Should |
| FR-10 | *Example:* Semantic cache, opt-in per route | Could |
| FR-11 | *Example:* Audit log of configuration changes and of access to stored prompts | Must |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Added latency | *Example:* p95 gateway overhead at or below 30 ms before the first streamed byte, excluding provider time | Span timing: gateway start to provider request sent |
| NFR-02 | Availability | *Example:* 99.95% monthly for the gateway itself | Successful admissions over valid requests |
| NFR-03 | End-to-end availability | *Example:* 99.9% for aliases with an approved fallback chain | Completed responses over admitted requests |
| NFR-04 | Quota accuracy | *Example:* Overshoot at or below 2% of a tenant's tokens per minute | Reconciliation of counters against usage records |
| NFR-05 | Metering completeness | *Example:* 100% of admitted requests produce a usage record within 5 minutes | Daily reconciliation against provider invoices |
| NFR-06 | Privacy | *Example:* No unredacted personal data reaches a provider not approved for it | Redaction test corpus; sampled audits |
| NFR-07 | Data retention | *Example:* Prompts and outputs not stored by default; opt-in logging with 30-day retention | Configuration audit |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* Running model servers and GPUs (see LLM Inference Serving and Vector Search Platform Design Template (`system-design/design-llm-inference-serving`))]
- [*Example:* Prompt design, retrieval and agent loops inside applications (see RAG System Design Template (`ai-data/rag-system-design`))]
- [*Example:* General API management for non-LLM traffic (see API Gateway System Design Template (`system-design/design-api-gateway`))]
- [*Example:* Content policy decisions themselves; the gateway runs the classifiers defined in Guardrails and Safety Filter Specification Template (`ai-data/guardrails-spec`)]
- [*Example:* Fine-tuning jobs and batch inference windows longer than 24 hours]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Every number must end in a consequence for the design. Show inputs, formulas and results; round generously and say which way. -->

### 3.1 Assumptions

All values are *Example* planning inputs.

| Input | Value | Source |
|---|---|---|
| Peak requests per second | 1,000 | Tenant forecasts |
| Average requests per second | 200 | Last 30 days of provider bills, converted |
| Mean input tokens per request | 1,200 | Sampled traffic |
| Mean output tokens per request | 300 | Sampled traffic |
| Mean stream duration | 6 s | Sampled traffic |
| Streams per gateway instance | 1,000 | Load test with redaction enabled |
| Usage record size (metadata only) | 1 KB | Schema in 5.1 |
| Logged payload size, when opted in | 8 KB | Prompt plus output plus envelope |
| Share of traffic with payload logging | 20% | Tenant opt-ins |
| Blended price, input | USD 2.50 per million tokens | *Example* contract |
| Blended price, output | USD 10 per million tokens | *Example* contract |

### 3.2 Concurrent streams and instances

Little's law gives concurrent streams = arrival rate x duration =
1,000 x 6 s = 6,000 open streams at peak. At 1,000 streams per instance the
gateway needs 6 instances at full load. Spread across three zones with four
instances each (12 in total), losing one zone leaves 8 instances with 8,000
stream capacity, so peak load runs at 75%. **Consequence:** the gateway must
use non-blocking I/O, and autoscaling must count open streams, not CPU, since
a waiting stream costs memory and a socket but almost no CPU.

### 3.3 Token rates

Peak input is 1,000 x 1,200 = 1.2 million tokens per second; peak output is
1,000 x 300 = 300,000 tokens per second. **Consequence:** request counts alone
cannot protect providers or budgets. Quotas must be in tokens, and the
gateway must reserve `input + max_tokens` at admission and settle the actual
count at the end (7.2).

### 3.4 Counter store load

Each request performs one reservation and one settlement, so the counter
store sees about 2,000 atomic operations per second at peak. **Consequence:**
one in-memory counter cluster per region is enough; key counters by tenant
and alias so a hot tenant lands on one shard and does not slow others.

### 3.5 Daily volume, storage and spend

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Requests per day | 200 x 86,400 | 17.28 million | Usage events go to a stream, never written synchronously to a relational database |
| Usage records per day | 17.28 million x 1 KB | 17.28 GB | Columnar store partitioned by day and tenant |
| Usage records, 400-day retention | 17.28 GB x 400 | about 6.9 TB | Retain 13 months for year-on-year chargeback, then aggregate |
| Logged payloads per day | 17.28 million x 0.2 x 8 KB | about 27.6 GB | Separate encrypted store, per-tenant keys |
| Logged payloads, 30 days | 27.6 GB x 30 | about 829 GB | Retention job is a release blocker, not a later task |
| Input tokens per day | 17.28 million x 1,200 | 20.736 billion | |
| Output tokens per day | 17.28 million x 300 | 5.184 billion | |
| Input spend per day | 20,736 x USD 2.50 | USD 51,840 | |
| Output spend per day | 5,184 x USD 10 | USD 51,840 | Output tokens cost as much as input at a quarter of the volume, so `max_tokens` caps matter |
| Spend per 30 days | (51,840 + 51,840) x 30 | about USD 3.11 million | Budgets and alerts per tenant are a core requirement, not a report |
| Saving from a 10% cache hit rate | 10% of USD 103,680 per day | about USD 10,368 per day | Exact cache is worth building first; semantic cache must prove quality before it is enabled |

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Open streams at peak | 6,000 | Non-blocking proxy; scale on open streams |
| Instances | 12 across 3 zones | Survives one zone loss at 75% load |
| Peak tokens per second | 1.2 million in, 300,000 out | Token-based quotas with reservation |
| Counter operations | about 2,000 per second | One counter cluster per region, keyed by tenant |
| Usage data | about 6.9 TB over 13 months | Stream plus columnar store |
| Spend | about USD 3.1 million per month | Budgets, alerts and chargeback from day one |

## 4. API design

<!-- Guidance: Define the contract first: resources, request and response shapes, streaming format, errors, idempotency and versioning. Use RFC 9457 problem details for errors. -->

### 4.1 Endpoints

| Method and path | Purpose |
|---|---|
| `POST /v1/chat/completions` | Chat completion; `stream: true` returns server-sent events |
| `POST /v1/embeddings` | Embed a batch of texts |
| `GET /v1/models` | Aliases the caller may use, with limits and data-policy labels |
| `GET /v1/usage?from=&to=&group_by=` | Tokens and cost per alias, route and day |
| `POST /admin/v1/routes` | Create or change a route (audited, change-managed) |
| `POST /admin/v1/tenants/{id}/quotas` | Set quotas and budgets (audited) |

### 4.2 Request example

The caller names an alias, never a provider. Gateway extensions travel in
headers so standard client libraries keep working.

```http
POST /v1/chat/completions
Authorization: Bearer [gateway key]
X-Use-Case: support-summary
X-Request-Id: 4f1c2e7a-0b9d-4c1e-9a55-2d7e8f0c6b11
X-Data-Class: personal
Content-Type: application/json

{"model": "chat-large", "stream": true, "max_tokens": 400, "messages": [{"role": "system", "content": "[system prompt]"}, {"role": "user", "content": "[ticket text]"}]}
```

Response headers report what actually happened:

| Header | Meaning |
|---|---|
| `X-Route` | Concrete route used, for example `provider-a/model-x@2026-08` |
| `X-Fallback` | `true` when a fallback route served the request |
| `X-Cache` | `hit-exact`, `hit-semantic` or `miss` |
| `X-Redacted` | Count of entities redacted before the provider call |

### 4.3 Errors

| Status | Problem type | Meaning |
|---|---|---|
| 400 | `context-too-long` | Input plus `max_tokens` exceeds the route's context window |
| 403 | `data-policy-denied` | Data class not allowed on any route of this alias |
| 422 | `guardrail-blocked` | Input or output blocked by a guardrail; category returned, content not echoed |
| 429 | `token-rate-exceeded` / `budget-exhausted` | Quota or monthly budget exhausted; `Retry-After` set for rate limits |
| 502 | `provider-error` | Provider failed and no fallback is approved |
| 503 | `capacity-unavailable` | All routes saturated or open-circuited |
| 504 | `deadline-exceeded` | Deadline passed before the first token |

Retries by callers carry the same `X-Request-Id`. The gateway deduplicates
non-streaming requests by this ID for 10 minutes (*Example*), so a retried
call after a network error is not billed twice. Streaming responses are not
replayed: a retried stream is a new, separately metered request, so clients
retry a stream only when no chunk was received.

## 5. Data model

<!-- Guidance: Choose storage by access pattern. List each query the system must answer and the key or index that serves it. -->

### 5.1 Entities

Configuration is small and audited; usage records are high-volume and
append-only. No entity stores prompt content unless the tenant opted in.

```mermaid
erDiagram
  TENANT ||--o{ GATEWAY_KEY : owns
  TENANT ||--o{ QUOTA : has
  TENANT ||--o{ DATA_POLICY : "bound by"
  ALIAS ||--o{ ROUTE : "resolves to"
  ROUTE }o--|| PROVIDER : uses
  TENANT ||--o{ USAGE_RECORD : generates
  TENANT ||--o{ AUDIT_EVENT : "subject of"
  TENANT {
    string tenant_id PK
    string cost_center
  }
  GATEWAY_KEY {
    string key_hash PK
    string tenant_id FK
    string use_case
    string status
  }
  QUOTA {
    string tenant_id PK
    string alias PK
    int tokens_per_minute
    int requests_per_minute
    decimal monthly_budget
  }
  ROUTE {
    string route_id PK
    string alias FK
    string provider_id FK
    string model_version
    int weight
    string region
    string allowed_data_classes
  }
  USAGE_RECORD {
    string request_id PK
    string tenant_id FK
    string route_id
    int input_tokens
    int output_tokens
    int cached_tokens
    decimal cost
  }
```

### 5.2 Access patterns

| ID | Query | Key or index that serves it | Partition key and rationale |
|---|---|---|---|
| AP-01 *(Example)* | Authenticate a key on every request | `key_hash` lookup, cached in gateway memory for 60 s | Small table, replicated to every instance |
| AP-02 *(Example)* | Read and update a tenant's token counter | Counter key `tenant:alias:minute` in the counter store | Tenant plus alias: spreads load, keeps one tenant's checks on one shard |
| AP-03 *(Example)* | Resolve an alias to eligible routes | In-memory routing table, pushed on change | Whole table fits in memory; versioned |
| AP-04 *(Example)* | Exact cache lookup | Hash of tenant, route, normalised request and parameters | Tenant in the key prevents cross-tenant hits |
| AP-05 *(Example)* | Monthly spend per tenant | Columnar table partitioned by day, clustered by tenant | Day partitions make retention a partition drop |
| AP-06 *(Example)* | Find all calls of one request ID in an incident | Trace store by trace ID; usage table by `request_id` | Request ID propagated as an attribute |
| AP-07 *(Example)* | Who changed a route and when | Append-only audit log by object and time | Write once, never updated |

### 5.3 Storage choices

| Data | Store | Notes |
|---|---|---|
| Tenants, keys, quotas, routes, policies | Relational database, cached in each instance | Changes propagate within 30 s; every change audited |
| Token counters | In-memory store with atomic scripts | Same pattern as Rate Limiter System Design Template (`system-design/design-rate-limiter`) |
| Exact cache | In-memory store with TTL | Values encrypted per tenant |
| Semantic cache | Vector index per tenant and route | Opt-in only |
| Usage records | Event stream to a columnar warehouse | Never on the request path |
| Logged payloads | Encrypted object storage with per-tenant keys | Off by default, 30-day retention |
| Audit events | Append-only log with immutable retention | 13 months |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request path and where state lives. Name components by responsibility. -->

```mermaid
flowchart LR
  app([Calling application]) --> edge[Gateway edge: TLS, auth]
  edge --> pol[Policy: quotas, data class, budget]
  pol --> red[PII redaction]
  red --> pre[Pre-call guardrail hooks]
  pre --> cache{Exact or semantic cache}
  cache -- hit --> resp[Response assembler]
  cache -- miss --> router[Router: alias to route, health, fallback]
  router --> a1[Adapter: provider A]
  router --> a2[Adapter: provider B]
  router --> a3[Adapter: self-hosted pool]
  a1 --> resp
  a2 --> resp
  a3 --> resp
  resp --> post[Post-call guardrail hooks, restoration]
  post --> app
  pol --- counters[(Token counters)]
  router --- cfg[(Routing table)]
  resp --> meter[[Usage and audit events]]
  meter --> wh[(Warehouse)]
  resp --> otel[[Traces and metrics]]
```

| Component | Responsibility | Scales by |
|---|---|---|
| Gateway edge | TLS, key authentication, request ID, deadline | Open streams |
| Policy | Quotas, budgets, data-class checks, route eligibility | With the edge (same process) |
| PII redaction | Detect and replace entities; keep a per-request mapping | CPU; model-based detectors on a side pool |
| Guardrail hooks | Call input and output classifiers per policy | Separate classifier pool |
| Cache | Exact and opt-in semantic lookup | Counter-store style cluster; vector index |
| Router | Choose a route by weight, health, region and policy; run fallback | In process, table pushed from config |
| Provider adapters | Translate requests, streams, tool calls and errors | In process; one module per provider |
| Metering | Emit usage and audit events; reconcile with invoices | Event stream |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Each deep dive lists options, the choice and the reason. -->

### 7.1 Request flow

```mermaid
sequenceDiagram
  participant A as App
  participant G as Gateway
  participant C as Counters
  participant R as Router
  participant P as Provider
  participant M as Metering
  A->>G: POST /chat/completions (stream)
  G->>G: authenticate key, resolve tenant and use case
  G->>C: reserve input + max_tokens
  C-->>G: ok (or 429)
  G->>G: redact PII, run pre-call hooks, check cache
  G->>R: eligible routes for alias and data class
  R->>P: provider request (adapter)
  P-->>G: stream chunks
  G-->>A: SSE chunks (restored entities, post-call hooks)
  P-->>G: final usage
  G->>C: settle actual tokens
  G->>M: usage record, trace
```

The reservation happens before any provider call, so a tenant over quota
never consumes provider capacity. Settlement uses the provider's reported
usage when present and the gateway's own count otherwise; both are kept in
the usage record so reconciliation can spot adapters that miscount.

### 7.2 Deep dive 1: quotas and budgets across tenants

| Option | Strengths | Weaknesses |
|---|---|---|
| A. Requests per minute only | Simple, cheap | One long prompt costs a hundred short ones; budgets drift |
| B. Tokens per minute with reservation of `input + max_tokens`, settled at the end | Protects providers and budgets; fair across prompt sizes | Large `max_tokens` holds quota during the stream; needs a tokenizer per model family |
| C. Post-hoc billing only, no admission control | No latency cost | Overspend discovered after the fact |

**Choice:** B, with requests per minute as a secondary guard and a monthly
currency budget checked at admission against spend so far.

**Reason:** spend scales with tokens (3.5), and providers enforce their own
token limits per account; mirroring them per tenant stops one tenant from
exhausting the shared provider account. Input tokens are counted with the
route's tokenizer at admission; when the route is unknown until fallback, the
gateway reserves using the largest tokenizer ratio of the eligible routes and
settles later.

| Rule | *Example* value |
|---|---|
| Counter window | Sliding window of 60 s in 10-s buckets |
| Overshoot allowed | Up to one in-flight reservation per tenant beyond the limit |
| Budget check | Spend so far plus reservation cost must stay under 100% of the monthly budget; alert at 80% |
| Burst | The whole per-minute quota may be used inside one 10-s bucket; no carry-over between windows |

### 7.3 Deep dive 2: routing and fallback

| Option | Strengths | Weaknesses |
|---|---|---|
| A. Static alias to one route | Predictable quality | Any provider outage is an outage for the alias |
| B. Weighted routes plus health-based failover inside one approved chain | Survives provider incidents; quality reviewed in advance | Needs evaluation of each route on the alias's tasks |
| C. Dynamic cost or latency optimiser across all models | Lowest cost | Quality changes without anyone deciding; can break data-policy rules |

**Choice:** B. Each alias has an ordered fallback chain whose routes all
passed the alias's evaluation suite and all satisfy the alias's data policy.

**Reason:** the 2 incidents in 1.1 came from fallback that changed quality or
data handling without approval. Fallback is a product decision, not a
networking one.

| Rule | Condition | Action |
|---|---|---|
| RT-01 *(Example)* | Alias `chat-large`, data class internal | 90% provider A, 10% provider B (live comparison) |
| RT-02 *(Example)* | Route error rate above 20% over 30 s, or 5 timeouts in a row | Open the circuit for 60 s; send traffic to the next route in the chain |
| RT-03 *(Example)* | Provider returns rate-limit errors | Back off with jitter; shift share to the next route; do not retry the same route more than once |
| RT-04 *(Example)* | Data class personal | Only routes with a data processing agreement and an approved region |
| RT-05 *(Example)* | First token not received before 50% of the deadline | Cancel and try the next route once; never after the first token was streamed |

A request is never retried on another route after the first token reached
the caller: the caller would see two different answers spliced together.
Retry rules and circuit breakers follow the Resilience and Load Shedding
Design Template (`system-design/resilience-load-shedding-design`).

### 7.4 Deep dive 3: PII redaction and restoration

| Option | Strengths | Weaknesses |
|---|---|---|
| A. Pattern rules only (emails, phone numbers, card numbers) | Fast, deterministic | Misses names and addresses |
| B. Rules plus a named-entity model, replacement with typed placeholders, mapping kept in memory for the request | Better recall; the answer can be restored for the caller | Adds latency; placeholders can confuse the model |
| C. Block requests with personal data | Simple guarantee | Blocks legitimate use cases |

**Choice:** B for routes to external providers; A for self-hosted routes in
the approved region; C only for data classes that no route may receive.

**Reason:** most use cases need names in the answer (a support summary must
name the customer), so redaction must be reversible inside the gateway while
the provider sees placeholders. The mapping lives only in request memory and
is never logged.

Rules:

- Redact before the cache lookup, so cache keys and cached values never
  contain raw personal data.
- Use typed, numbered placeholders (`[PERSON_1]`, `[EMAIL_1]`); keep the
  numbering stable within one request so the model can refer back.
- Restore on the streamed output; buffer partial placeholders that split
  across chunks before writing to the caller.
- Measure recall on a labelled corpus per language; NFR-06 depends on it.

> **How real systems do it: Uber's GenAI Gateway (paraphrased)**
>
> Uber's engineering blog describes a gateway, written in Go, that gives
> internal teams one interface to third-party and in-house models and
> mirrors the OpenAI API so existing client libraries work. Before a request
> leaves for an external vendor, a redaction step swaps detected personal
> data for numbered placeholders and keeps the mapping, which the gateway
> then uses to put the original values back into the vendor's answer. Audit
> logs support cost attribution, security audits and quality evaluation, and
> each use case passes a security review against the company's data handling
> standard before it gets access. Source: Wang and Bansal, "Navigating the
> LLM Landscape: Uber's Innovation with GenAI Gateway", Uber Engineering
> blog, July 2024. Read the original for details; this box is a summary.

### 7.5 Deep dive 4: exact and semantic caching

| Option | Strengths | Weaknesses |
|---|---|---|
| A. No cache | No staleness, no leakage risk | Pays for every repeated call |
| B. Exact cache on normalised request plus parameters, only when temperature is 0 or a seed is fixed | Safe, cheap | Low hit rate on free-text traffic |
| C. Semantic cache: embed the last user turn, return a cached answer above a similarity threshold | Higher hit rate | Can return an answer to a different question; needs per-route evaluation |
| D. Provider-side prompt caching of long shared prefixes | Cuts input cost and time to first token | Provider-specific; counts must be metered separately |

**Choice:** B on by default for eligible requests; D passed through and
metered as cached tokens; C opt-in per route after an evaluation shows the
wrong-answer rate below the route's threshold.

**Reason:** a semantic hit that answers a different question is a quality
incident, not a cost saving. Cache entries are scoped to the tenant and the
route version and expire when the route's model version changes.

### 7.6 Streaming and provider adapters

- The gateway streams to callers with server-sent events (`text/event-stream`,
  WHATWG HTML Living Standard section 9.2) in one unified chunk format, and
  converts each provider's streaming events to it, including tool-call
  deltas and the final usage chunk.
- Adapters normalise finish reasons, error codes and token counts into the
  gateway's vocabulary, and keep a test suite of recorded provider
  responses so a provider API change is caught in CI.
- Client disconnect cancels the provider request and settles the tokens
  generated so far.

### 7.7 Guardrail hooks

| Hook | Position | Mode | *Example* checks |
|---|---|---|---|
| HK-01 | Before cache and provider | Synchronous | Prompt injection classifier on untrusted content, blocked topics |
| HK-02 | On streamed output, every N tokens | Synchronous, buffered | Sensitive data leakage, policy categories |
| HK-03 | After completion | Asynchronous | Quality and safety sampling for evaluation |

Hook definitions and thresholds live in the Guardrails and Safety Filter
Specification Template (`ai-data/guardrails-spec`); the gateway only runs
them and records the decision.

### 7.8 Telemetry and audit

Emit one span per model call using the OpenTelemetry semantic conventions for
generative AI, which have **Development** status and are now maintained in the
`open-telemetry/semantic-conventions-genai` repository. Pin the convention
version you implement and record it in Document control.

| Attribute | Use |
|---|---|
| `gen_ai.operation.name` | `chat` or `embeddings` |
| `gen_ai.provider.name` | Provider of the chosen route |
| `gen_ai.request.model`, `gen_ai.response.model` | Requested and served model |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | Token counts for metering cross-checks |
| `error.type` | Normalised error class (stable attribute) |
| Gateway-specific (namespaced) | Tenant, alias, route, fallback flag, cache result, redaction count |

Prompt and output content are not recorded on spans by default. Full
instrumentation rules are in the LLM and Agent Observability Specification
Template (`ai-data/llm-observability-spec`).

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it breaks first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Open streams | Non-blocking proxy, scale on open streams | Memory per stream (buffers for restoration) |
| Counter store | Shard by tenant and alias; local pre-aggregation for very hot tenants | Single hot key above about 50,000 operations per second (*Example*) |
| Redaction latency | Rules in process; entity model on a side pool with a 15 ms budget | Long prompts: redact in chunks in parallel |
| Routing table size | In memory, pushed on change | Tens of thousands of routes |
| Provider limits | Per-account limits mirrored as global quotas | Contracted provider limits |

Latency budget for gateway overhead (target p95 30 ms):

| Step | Budget |
|---|---|
| Authentication and policy (cached) | 2 ms |
| Counter reservation | 3 ms |
| Redaction (rules plus model) | 15 ms |
| Pre-call hook (fast classifier) | 8 ms |
| Cache lookup and routing | 2 ms |
| Headroom | 0 ms: the steps run in sequence, so the redaction model is the first step to cut when the budget is missed |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| ID | Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|---|
| FM-01 | Provider A | Regional outage or error storm | Route error rate, circuit state | Aliases routed to A | Fallback chain (7.3); status page for tenants | Under 1 min (automatic) |
| FM-02 | Counter store | Unavailable | Reservation errors | All tenants' quota checks | Fail open with a local per-instance cap for 5 min, then fail closed | 5 min |
| FM-03 | Redaction model pool | Slow or down | Hook latency, errors | Requests with personal data to external routes | Fail closed for personal data; rules-only for internal routes | 10 min |
| FM-04 | Routing config push | Bad route published | Canary error rate, schema validation | One alias or all | Versioned table, staged rollout, one-click rollback | 5 min |
| FM-05 | Metering stream | Backlog or loss | Lag metric, reconciliation gap | Chargeback accuracy, not traffic | Durable stream with replay; local spool on instances | 1 h |
| FM-06 | Gateway region | Region loss | Health checks | All callers in region | Active-active regions; DNS or anycast failover | 15 min |
| [FM-nn] | [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security

Mapping to the OWASP Top 10 for LLM Applications (2026 edition) and related
controls; confirm entry numbering against the current list at genai.owasp.org.

| ID | Threat | Control |
|---|---|---|
| SEC-01 | Leaked gateway key used by an outsider | Keys bound to use case and network zone; hashed at rest; rotation and anomaly alerts |
| SEC-02 | Provider credential exposure | Credentials only in the gateway's secret store; never returned or logged |
| SEC-03 | Sensitive Information Disclosure (LLM02) through prompts sent to providers | Data-class policy, redaction, approved regions |
| SEC-04 | Unbounded Consumption (LLM06), denial of wallet | Token reservation, budgets, `max_tokens` caps per alias |
| SEC-05 | Prompt Injection (LLM01) in untrusted content | Pre-call hook HK-01; application controls remain the primary defence |
| SEC-06 | Cache leakage across tenants | Tenant in every cache key; per-tenant encryption |
| SEC-07 | Insider reads logged prompts | Opt-in logging, access audited, break-glass approval |
| [SEC-nn] | [Threat] | [Control] |

Run a full threat model with the Threat Model Template (`security/threat-model`).

### 9.3 Observability

| Signal | Metric | Alert threshold |
|---|---|---|
| Gateway overhead | p95 per step | Above 30 ms for 10 min |
| Time to first token | p95 per alias and route | Above the alias SLO for 10 min |
| Route health | Error rate, circuit state | Circuit open for more than 5 min |
| Fallback share | Percentage of requests per alias | Above 10% for 15 min |
| Quota denials | 429 per tenant | Sudden rise per tenant |
| Budget | Spend versus budget | 80% and 100% of monthly budget |
| Metering reconciliation | Gateway tokens versus provider invoice | Gap above 1% per month |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Provider tokens | Contract prices | From 3.5 | [USD] |
| Gateway instances | [USD per instance-month] | 12 | [USD] |
| Counter and cache clusters | [USD per node-month] | [Nodes] | [USD] |
| Redaction and classifier pools | [USD per node-month] | [Nodes] | [USD] |
| Warehouse and logs | [USD per TB-month] | about 7.7 TB | [USD] |
| **Total** | | | **[USD]** |

Track spend with the Cloud Cost Review Template (FinOps) (`ops/finops-cost-review`).

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list options, the choice, why, and what you give up. Add what changes at 10x and 0.1x scale. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Build or buy | Open-source gateway; managed gateway; in-house | [In-house on an open-source proxy] | Data policy and metering are company-specific | Maintenance of adapters |
| API shape | Own schema; OpenAI-compatible | OpenAI-compatible | Existing libraries work unchanged | Provider-specific features need extensions |
| Quota unit | Requests; tokens; currency | Tokens plus currency budget | Matches cost and provider limits | Tokenizer per model family |
| Fallback | None; approved chains; optimiser | Approved chains | Quality and data policy stay decided | Some cost savings |
| Redaction place | Application; gateway | Gateway | One control, one test corpus | Gateway latency and CPU |
| Semantic cache | Off; on | Off by default, opt-in | Wrong-answer risk | Hit rate |
| What changes at 10x (10,000 requests per second at peak) | | Shard counters by tenant hash; regional gateways with local quotas and periodic global reconciliation; dedicated redaction fleet | One counter cluster and shared redaction pool become the bottleneck | Quota accuracy across regions becomes eventual |
| What changes at 0.1x (100 requests per second at peak) | | Single regional deployment of 3 instances; managed counter store; rules-only redaction for internal routes | Fixed cost of a multi-region fleet is not justified | Lower availability and redaction recall on internal routes |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Edge, keys, one provider adapter, metering | Two tenants migrated; usage reconciles within 1% of the invoice | YYYY-MM-DD |
| M2 | Token quotas, budgets, second provider, fallback chains | Failover drill meets NFR-03 | YYYY-MM-DD |
| M3 | Redaction and restoration, guardrail hooks | Redaction recall target met per language; NFR-06 audit passed | YYYY-MM-DD |
| M4 | Exact cache, self-hosted route | Hit rate and saving reported per alias | YYYY-MM-DD |
| M5 | Provider keys removed from all applications | Secret scan shows no provider keys outside the gateway | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Tenants move one use case at a time by changing their base URL;
direct provider access remains for 30 days and is then revoked. Route changes
roll out to 5% of an alias's traffic first and roll back automatically on
error-rate breach.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Gateway becomes a single point of failure | Medium | High | Multi-zone, multi-region, fail-open rules for counters | [@name] |
| R-02 | *Example:* Adapters lag behind new provider features | High | Medium | Passthrough extension field; adapter test suite | [@name] |
| R-03 | *Example:* Redaction misses entities in some languages | Medium | High | Per-language corpus; block personal data where recall is unproven | [@name] |
| R-04 | *Example:* Teams bypass the gateway | Medium | Medium | Egress rules block provider endpoints from application networks | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Traceability, open questions, glossary and references. -->

### 12.1 Traceability

| Requirement | Design element | Verification |
|---|---|---|
| FR-04, NFR-04 | 7.2 quotas and budgets | Load test with mixed prompt sizes; reconciliation |
| FR-05, NFR-03 | 7.3 routing and fallback | Provider outage drill |
| FR-06, NFR-06 | 7.4 redaction | Labelled corpus per language; sampled audit |
| FR-08, NFR-05 | 7.1 settlement, 5.3 metering | Monthly invoice reconciliation |
| NFR-01 | 8 latency budget | Span timing in production |
| [ID] | [Element] | [Test] |

### 12.2 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do batch APIs of providers go through the gateway or a separate path? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Which telemetry convention version do we pin while the GenAI conventions are in Development? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Alias | Stable model name used by callers, mapped by the gateway to routes |
| Route | Concrete provider, model version and region that can serve an alias |
| Fallback chain | Ordered list of approved routes tried when the preferred route fails |
| Redaction | Replacing detected personal data with placeholders before a provider call |
| Semantic cache | Cache that returns a stored answer for a similar, not identical, request |
| Denial of wallet | Attack or bug that drives model spend instead of taking a service down |

### 12.4 References

| Reference | Use |
|---|---|
| OpenTelemetry semantic conventions for generative AI (Development status), https://github.com/open-telemetry/semantic-conventions-genai | Span and metric names in 7.8 |
| OWASP Top 10 for LLM Applications, 2026 edition, OWASP GenAI Security Project, https://genai.owasp.org/ (CC BY-SA 4.0) | Threat names in 9.2 |
| WHATWG HTML Living Standard, section 9.2 Server-sent events | Streaming format in 7.6 |
| RFC 9457, Problem Details for HTTP APIs (2023) | Error format in 4.3 |
| Wang and Bansal, "Navigating the LLM Landscape: Uber's Innovation with GenAI Gateway", Uber Engineering blog, 11 July 2024 | Box in 7.4 (paraphrased) |
| LLM Inference Serving and Vector Search Platform Design Template (`system-design/design-llm-inference-serving`) | Self-hosted routes |
| API Gateway System Design Template (`system-design/design-api-gateway`) | General API management |
| Rate Limiter System Design Template (`system-design/design-rate-limiter`) | Counter algorithms |
| LLM and Agent Observability Specification Template (`ai-data/llm-observability-spec`) | Instrumentation rules |
| Guardrails and Safety Filter Specification Template (`ai-data/guardrails-spec`) | Hook definitions |
| Agentic AI Threat Model Template (`ai-data/agentic-threat-model`) | Threats when agents call the gateway |

## Standard and licence note

Based on the OpenTelemetry semantic conventions for generative AI
(Development status, Apache-2.0), the WHATWG HTML Living Standard section on
server-sent events, RFC 9457, and the OWASP Top 10 for LLM Applications 2026
by the OWASP GenAI Security Project (CC BY-SA 4.0; OWASP entry names are
attributed to OWASP, and any adapted OWASP material you redistribute must
carry the same licence under its ShareAlike term). The engineering write-up
in 7.4 is paraphrased and cited as inspiration only. The text is original;
provider behaviour, prices and convention names change often, so check the
current official version of each source before approval. Template licence:
MIT.