# System Design: [LLM inference platform name]

> Reference design for a shared platform that serves large language models
> and vector search to internal product teams. It covers the request flow
> (gateway, router, model servers), continuous batching, the KV cache and
> paged attention, prefill versus decode, GPU memory sizing, latency SLOs
> (time to first token and time per output token), GPU autoscaling, model
> routing and fallback, quantization, multi-tenancy and rate limits, the
> vector index (HNSW and IVF), the embedding pipeline and cost per million
> tokens. It is the platform underneath applications such as the one in the
> RAG System Design Template (`ai-data/rag-system-design`). It follows the
> structure of the System Design Document template and is pre-filled with a
> worked design to adapt. Replace every `[bracketed]` value and every row
> marked *Example*; all numbers are illustrative examples, not benchmarks.

<!--
Guidance: How to use this template
- Start from the workload, not the GPU: prompt length, output length,
  requests per second and latency targets per tenant (2.2 and 3.1). Serving
  cost is set almost entirely by these four numbers.
- Size memory before compute. On most deployments the KV cache, not the
  weights, decides how many requests run concurrently (3.3).
- Treat prefill and decode as two different workloads: prefill is
  compute-bound and sets time to first token; decode is memory-bandwidth-bound
  and sets time per output token.
- Re-measure every example number on your hardware and serving engine before
  approval. Vendor peak figures are upper bounds, not planning values.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-LLM-001] |
| System / feature | [LLM inference and vector search platform] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Models in scope | [Model names, versions and licences] |
| Related | [RAG system design], [LLM observability spec], [Capacity plan], [AI risk assessment] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead, ML platform | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security and privacy | [@name] | [Decision] | YYYY-MM-DD |
| FinOps / budget owner | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added FP8 variant and routing of short prompts to the small model |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Replaced flat vector index with HNSW plus scalar quantization |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Six product teams each run their own model server on dedicated
GPUs. Average GPU utilisation is 18%, yet two teams hit out-of-memory errors
at peak because long prompts exhaust the KV cache. There is no shared rate
limiting, so one batch job raised time to first token for the support
assistant from 0.6 s to 9 s. Each team also runs its own vector database,
with recall nobody has measured.]

### 1.2 Proposed solution

[*Example:* A single platform behind an OpenAI-compatible gateway. The
gateway authenticates tenants, applies token-based rate limits and passes
requests to a router that picks a model pool by model name, prompt length,
priority class and pool health. Each pool runs replicas of an inference
engine with continuous batching and a paged KV cache (in the style of vLLM
PagedAttention). Pools autoscale on queue depth and KV cache usage, not CPU.
A separate retrieval service offers embedding and vector search over HNSW
indexes per collection. Usage is metered per tenant in tokens and converted
to cost per million tokens for chargeback.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* One shared fleet or per-team GPUs? | Shared pools per model, isolation by priority class and quotas | 7.8, 10 |
| Q-02 | *Example:* Precision for the large model? | BF16 weights at launch; FP8 pool after evaluation parity is shown | 7.7 |
| Q-03 | *Example:* Autoscaling signal? | Pending requests and KV cache usage per replica | 7.6 |
| Q-04 | *Example:* Fallback when the large pool is saturated? | Route low-priority traffic to the small model; never silently for high priority | 7.5 |
| Q-05 | *Example:* Vector index type? | HNSW with int8 scalar quantization; IVF-PQ for the archive collection | 7.9 |
| [Q-nn] | [Question] | [Option] | [Section] |

## 2. Requirements

<!-- Guidance: Functional requirements describe behaviour; non-functional requirements describe qualities with numbers. Mark what is out of scope so nobody designs for it. -->

### 2.1 Functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Chat and completion endpoints with streaming responses | Must |
| FR-02 | *Example:* Serve at least two model sizes (large 70B-class, small 8B-class) | Must |
| FR-03 | *Example:* Per-tenant API keys, quotas in tokens per minute and requests per minute | Must |
| FR-04 | *Example:* Priority classes: interactive, standard, batch | Must |
| FR-05 | *Example:* Embedding endpoint and vector search with metadata filters | Must |
| FR-06 | *Example:* Usage metering per tenant, model and priority class | Must |
| FR-07 | *Example:* Model version pinning and gradual rollout of new versions | Should |
| FR-08 | *Example:* Prefix caching for shared system prompts | Should |
| FR-09 | *Example:* Offline batch inference with a 24-hour completion window | Could |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Time to first token (TTFT), interactive | *Example:* p95 at or below 1.0 s for prompts up to 4 000 tokens | Gateway timing, first streamed byte |
| NFR-02 | Time per output token (TPOT), interactive | *Example:* p95 at or below 50 ms | Inter-token interval at the gateway |
| NFR-03 | Availability | *Example:* 99.9% monthly for interactive class | SLO on successful completions |
| NFR-04 | Isolation | *Example:* Batch load cannot raise interactive TTFT p95 by more than 20% | Load test with mixed tenants |
| NFR-05 | Vector search | *Example:* recall at 10 at or above 0.95, p99 at or below 50 ms at 2 000 queries/s | Recall test set; service timing |
| NFR-06 | Cost | *Example:* Within budget per million tokens in 3.6 | Monthly chargeback report |
| NFR-07 | Data handling | *Example:* Prompts and outputs not stored by default; tenant opt-in logging with 30-day retention | Configuration audit |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Out of scope

- [*Example:* model training and fine-tuning pipelines]
- [*Example:* application-level retrieval logic, prompt design and evaluation (see RAG System Design Template (`ai-data/rag-system-design`) and LLM Evaluation Plan Template (`ai-data/llm-evaluation-plan`))]
- [*Example:* calls to external hosted model APIs (handled by the gateway as a passthrough, not served here)]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. -->

### 3.1 Assumptions

All values are *Example* planning inputs for the large model pool. Replace
them with measurements from your engine and hardware.

| Input | Symbol | Value | Source |
|---|---|---|---|
| Parameters | $P$ | 70 billion | Model card |
| Transformer layers | $L$ | 80 | Model config |
| KV heads (grouped-query attention) | $h_{kv}$ | 8 | Model config |
| Head dimension | $d_h$ | 128 | Model config |
| Bytes per value (BF16) | $b$ | 2 | Precision choice |
| GPUs per replica (tensor parallel) | $g$ | 4 | 7.2 |
| Memory per GPU | $m$ | 80 GB | Hardware |
| HBM bandwidth per GPU | $B_w$ | 3.35 TB/s | Hardware datasheet (peak) |
| Dense BF16 compute per GPU | $F$ | 1 000 TFLOP/s | Rounded datasheet peak |
| Model FLOP utilisation in prefill | $\eta_p$ | 0.4 | Engine benchmark |
| Bandwidth efficiency in decode | $\eta_d$ | 0.6 | Engine benchmark |
| Peak requests per second | $R$ | 20 | Tenant forecasts |
| Mean prompt tokens | $n_{in}$ | 1 500 | Tenant logs |
| Mean output tokens | $n_{out}$ | 500 | Tenant logs |
| Maximum batch (sequences per step) | $B$ | 128 | Engine setting |
| GPU cost per hour | $c_g$ | USD 2.50 | *Example* contract price |

### 3.2 Weights

$$
M_{w} = P \times b = 70 \times 10^{9} \times 2 \ \text{B} = 140 \ \text{GB}
$$

The weights do not fit one 80 GB GPU, so one replica spans $g = 4$ GPUs
(320 GB in total).

### 3.3 KV cache per token and concurrency

Each layer stores one key and one value vector per KV head per token:

$$
s_{kv} = 2 \times L \times h_{kv} \times d_h \times b = 2 \times 80 \times 8 \times 128 \times 2 = 327\,680 \ \text{B} \approx 320 \ \text{KiB per token}
$$

The engine reserves 90% of GPU memory and keeps 8 GB for activations and
runtime buffers, leaving for the KV cache pool:

$$
M_{kv} = 0.9 \times 320 - 140 - 8 = 140 \ \text{GB}, \qquad
N_{tok} = \frac{140 \times 10^{9}}{327\,680} \approx 427\,000 \ \text{tokens}
$$

During decode a sequence holds its prompt plus, on average, half its output,
so the mean resident length is $1\,500 + 250 = 1\,750$ tokens. A full batch of
128 sequences uses

$$
128 \times 1\,750 \times 327\,680 \ \text{B} \approx 73.4 \ \text{GB}
$$

which fits the 140 GB pool with room for long prompts. The pool could hold
about $427\,000 / 1\,750 \approx 244$ average sequences, so with $B = 128$ the
batch limit, not memory, binds for average traffic. Long prompts change that:
128 sequences at the 8 192-token maximum would need about 344 GB, so the
scheduler must queue or preempt (7.3).

### 3.4 Decode and prefill rates per replica

Decode reads all weights plus the batch's KV cache once per step:

$$
t_{step} \approx \frac{M_w + M_{kv,batch}}{g \times B_w \times \eta_d} = \frac{140 + 73.4 \ \text{GB}}{4 \times 3.35 \ \text{TB/s} \times 0.6} \approx 26.5 \ \text{ms}
$$

So TPOT is about 27 ms (within NFR-02) and decode throughput per replica is

$$
D = \frac{B}{t_{step}} = \frac{128}{0.0265 \ \text{s}} \approx 4\,800 \ \text{output tokens/s}
$$

Prefill costs about $2P$ floating-point operations per prompt token:

$$
F_{rep} = g \times F \times \eta_p = 4 \times 10^{15} \times 0.4 = 1.6 \times 10^{15} \ \text{FLOP/s}, \qquad
P_{rate} = \frac{1.6 \times 10^{15}}{2 \times 70 \times 10^{9}} \approx 11\,400 \ \text{prompt tokens/s}
$$

A 1 500-token prompt alone takes about $1\,500 / 11\,400 \approx 0.13$ s to
prefill, so the 1 s TTFT target leaves most of the budget for queueing and
batching delay.

### 3.5 Replicas for peak load

Peak demand is $R \times n_{in} = 30\,000$ prompt tokens/s and
$R \times n_{out} = 10\,000$ output tokens/s. Treating prefill and decode time
as shares of one replica:

$$
U = \frac{30\,000}{11\,400} + \frac{10\,000}{4\,800} \approx 2.63 + 2.08 = 4.71 \ \text{replica-equivalents}
$$

At a target utilisation of 0.7 to protect latency, $4.71 / 0.7 \approx 6.7$,
rounded up to 7 replicas, plus one for failure and rollout (N+1): 8 replicas,
or 32 GPUs at peak.

### 3.6 Cost per million tokens

At peak the pool costs $32 \times 2.50 = 80$ USD per hour and serves
$30\,000 \times 3\,600 = 108$ million prompt tokens and
$10\,000 \times 3\,600 = 36$ million output tokens per hour. Splitting cost by
the replica time each phase uses (2.63 and 2.08 of 4.71, so 56% and 44%):

$$
c_{in} = \frac{0.56 \times 80}{108} \approx 0.41 \ \text{USD per million prompt tokens}, \qquad
c_{out} = \frac{0.44 \times 80}{36} \approx 0.98 \ \text{USD per million output tokens}
$$

These are peak-hour figures. Off-peak capacity that cannot be scaled down
raises the daily average; publish the monthly figure from metering, not
this estimate.

### 3.7 Vector index memory

| Item | Calculation | Result |
|---|---|---|
| Chunks indexed | *Example:* 100 million | |
| Raw float32 vectors, 768 dimensions | 100 000 000 x 768 x 4 B | 307.2 GB |
| HNSW graph links, $M = 32$ (64 links on the base layer, 4 B each) | 100 000 000 x 64 x 4 B | about 25.6 GB |
| int8 scalar quantization | 100 000 000 x 768 x 1 B | 76.8 GB |
| Product quantization, 96 sub-vectors of 8 bits | 100 000 000 x 96 B | 9.6 GB |
| Embedding ingest | *Example:* 5 million new chunks/day x 400 tokens | 2 billion tokens/day, about 23 000 tokens/s on average |

### 3.8 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Large model replica | 4 GPUs, 140 GB weights, 140 GB KV pool | Tensor parallel within one node |
| Concurrency per replica | Batch of 128; about 244 average sequences fit in memory | Batch limit set by latency, preemption for long prompts |
| TPOT | About 27 ms | Meets 50 ms target with headroom |
| Peak replicas | 8 (32 GPUs) | Autoscale between minimum 3 and maximum 10 |
| Cost at peak | About USD 0.41 per million prompt tokens and USD 0.98 per million output tokens | Chargeback per tenant |
| Vector index | About 100 GB per 100 million chunks with int8 plus graph | Memory-optimised nodes, sharded by collection |

## 4. API design

<!-- Guidance: Define the contract first: resources, methods, request and response shapes, errors, idempotency, pagination, rate limits and versioning. Use RFC 9457 problem details for errors. -->

### 4.1 Endpoints

| Method and path | Purpose |
|---|---|
| `POST /v1/chat/completions` | Chat completion, streaming with server-sent events |
| `POST /v1/embeddings` | Embed a batch of texts |
| `POST /v1/collections/{id}/query` | Vector search with filters and top-k |
| `POST /v1/collections/{id}/upsert` | Add or replace vectors with metadata |
| `POST /v1/batches` | Submit offline batch inference |
| `GET /v1/models` | List models, versions and limits available to the tenant |
| `GET /v1/usage?from=&to=` | Tokens and cost per model and class |

### 4.2 Request example

The tenant names a model alias, not a pool. The router maps the alias to a
pinned version and pool; the response reports the version actually used.

```http
POST /v1/chat/completions
Authorization: Bearer [tenant key]
X-Priority: interactive
Content-Type: application/json

{"model": "assistant-large", "stream": true, "max_tokens": 600, "messages": [{"role": "system", "content": "[system prompt]"}, {"role": "user", "content": "[question]"}]}
```

### 4.3 Limits and errors

| Status | Problem type | Meaning |
|---|---|---|
| 400 | `context-too-long` | Prompt plus `max_tokens` exceeds the model's context window |
| 429 | `token-rate-exceeded` | Tenant tokens per minute exhausted; `Retry-After` set |
| 503 | `capacity-unavailable` | Pool saturated and no permitted fallback; `Retry-After` set |
| 504 | `deadline-exceeded` | Request deadline passed before first token |

Rate limits count both prompt and output tokens. The gateway reserves
`prompt_tokens + max_tokens` at admission and refunds the unused part on
completion, so large `max_tokens` values cost quota until the stream ends.

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

Configuration entities (tenants, quotas, models, pools) are small and
audited; usage records and vectors are high-volume and partitioned.

```mermaid
erDiagram
  TENANT ||--o{ API_KEY : owns
  TENANT ||--o{ QUOTA : has
  MODEL ||--o{ MODEL_VERSION : has
  MODEL_VERSION ||--o{ POOL : "served by"
  TENANT ||--o{ USAGE_RECORD : generates
  TENANT ||--o{ COLLECTION : owns
  COLLECTION ||--o{ VECTOR_RECORD : contains
  TENANT {
    string tenant_id PK
    string default_priority
  }
  QUOTA {
    string tenant_id PK
    string model_alias PK
    int tokens_per_minute
    int requests_per_minute
  }
  MODEL_VERSION {
    string version_id PK
    string precision
    int context_window
  }
  POOL {
    string pool_id PK
    string version_id FK
    int min_replicas
    int max_replicas
  }
  USAGE_RECORD {
    string request_id PK
    string tenant_id FK
    int prompt_tokens
    int output_tokens
    int cached_tokens
  }
  COLLECTION {
    string collection_id PK
    string embedding_model
    string index_type
  }
  VECTOR_RECORD {
    string vector_id PK
    string collection_id FK
    string metadata
  }
```

### 5.2 Storage choices

| Data | Store | Notes |
|---|---|---|
| Tenants, keys, quotas, routing table | Relational database, cached in the gateway | Changes propagate within 30 s |
| Rate limit counters | In-memory store with atomic scripts | Same pattern as Rate Limiter System Design (`system-design/design-rate-limiter`) |
| Usage records | Event stream to the analytics warehouse | Never on the request path |
| Model weights | Object storage plus node-local cache | Pre-pulled on node start to cut cold start |
| Vector indexes | Vector database nodes, sharded by collection | Snapshots to object storage |
| Prompt and output logs | Off by default; tenant opt-in store with retention policy | See NFR-07 |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

```mermaid
flowchart LR
  app([Tenant application]) --> gw[Inference gateway]
  gw --> auth[Auth, quotas, token rate limits]
  auth --> router[Model router]
  router --> q1[[Queue: large pool]]
  router --> q2[[Queue: small pool]]
  q1 --> large[Large model replicas, 4 GPUs each]
  q2 --> small[Small model replicas, 1 GPU each]
  large --> kv[(Paged KV cache in GPU memory)]
  gw --> ret[Retrieval service]
  ret --> emb[Embedding replicas]
  ret --> vdb[(Vector index shards)]
  ingest[[Document ingest]] --> embq[[Embedding queue]]
  embq --> emb
  emb --> vdb
  large --> metrics[[Metrics and usage events]]
  small --> metrics
  metrics --> scaler[GPU autoscaler]
  scaler --> large
  scaler --> small
```

| Component | Responsibility | Scales by |
|---|---|---|
| Inference gateway | Auth, quotas, streaming proxy, metering | Horizontally (CPU) |
| Model router | Map alias to pool; choose by length, priority, health; fallback | Horizontally |
| Pool queues | Per-pool, per-priority queues with deadlines | One per pool |
| Model replicas | Inference engine with continuous batching and paged KV cache | GPUs, per pool |
| Retrieval service | Query embedding, vector search, filters | Horizontally plus index shards |
| Embedding replicas | Batch embedding for queries and ingest | GPUs, separate from chat pools |
| GPU autoscaler | Add or remove replicas on queue and KV signals | Control loop |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

### 7.1 Request flow

One streaming chat request, from admission to metering. The quota
reservation happens before queueing so a request that will exceed the
tenant limit never occupies GPU memory.

```mermaid
sequenceDiagram
  participant A as App
  participant G as Gateway
  participant R as Router
  participant Q as Pool queue
  participant M as Model replica
  A->>G: POST /chat/completions (stream)
  G->>G: authenticate, reserve prompt plus max tokens
  G->>R: route (alias, prompt length, priority)
  R->>Q: enqueue with deadline
  Q->>M: admit when KV blocks available
  M->>M: prefill prompt (populate KV cache)
  M-->>G: first token
  G-->>A: first token (TTFT)
  loop each decode step
    M-->>G: next token
    G-->>A: stream
  end
  M-->>G: done, token counts
  G->>G: refund unused reservation, emit usage
```

### 7.2 Model servers and parallelism

- **Tensor parallelism** splits each layer across the GPUs of one node; it
  needs fast GPU-to-GPU links, so a replica stays inside one node.
- **Pipeline parallelism** splits layers across nodes; use it only for models
  that do not fit one node, because it adds latency per step.
- **Data parallelism** is the replica count; it is what the autoscaler
  changes.
- *Example:* the large pool runs 4-way tensor parallel per replica; the small
  model runs one replica per GPU.

### 7.3 Continuous batching, KV cache and paged attention

Static batching waits for every sequence in a batch to finish, so short
answers wait for long ones and GPUs idle. Continuous (iteration-level)
batching, described for the Orca serving system (Yu et al., OSDI 2022), adds
and removes sequences at every decode step, so the batch stays full.

The KV cache stores keys and values of every past token so each new token
attends to them without recomputation. Reserving contiguous memory for each
sequence's maximum length wastes most of it. PagedAttention (Kwon et al.,
"Efficient Memory Management for Large Language Model Serving with
PagedAttention", SOSP 2023) stores the KV cache in fixed-size blocks mapped
through a per-sequence block table, like virtual memory pages. Blocks are
allocated as a sequence grows, freed when it ends and can be shared between
sequences with a common prefix.

Design consequences:

| Concern | Rule |
|---|---|
| Block size | *Example:* 16 tokens per block, so the 427 000-token pool holds about 26 700 blocks |
| Admission | Admit a sequence only when its prompt's blocks are free |
| Preemption | When blocks run out mid-decode, preempt the lowest-priority, most recent sequence (swap to host memory or recompute later) |
| Prefix caching | Share blocks for identical system prompts; count cached tokens separately in metering |
| Context limit | Enforce per-pool maximum context so one request cannot occupy the whole pool |

### 7.4 Prefill versus decode

| Property | Prefill | Decode |
|---|---|---|
| Work | All prompt tokens in parallel | One token per sequence per step |
| Bound by | Compute | Memory bandwidth |
| Sets | TTFT | TPOT |
| Interference | A long prefill pauses decode steps of the batch | Large batches raise TPOT slowly |

Mitigations: split long prompts into chunks interleaved with decode steps
(chunked prefill) so decode latency stays bounded; cap prompt tokens
prefilled per step; for very long-context tenants consider a separate pool,
or disaggregated prefill and decode replicas once the engine supports it.

### 7.5 Model routing and fallback

| Rule | Condition | Action |
|---|---|---|
| RT-01 *(Example)* | Alias `assistant-large`, interactive | Large pool |
| RT-02 *(Example)* | Prompt over 16 000 tokens | Long-context pool |
| RT-03 *(Example)* | Large pool queue wait p95 above 2 s, priority batch | Defer batch until queue clears |
| RT-04 *(Example)* | Large pool unhealthy, tenant opted in to fallback | Small pool, response header `X-Model-Fallback: true` |
| RT-05 *(Example)* | New model version in canary | 5% of traffic of opted-in tenants |

Fallback to a different model changes answer quality, so it is opt-in per
tenant and always visible in the response and in metering.

### 7.6 Autoscaling on GPUs

- Scale on pending requests per replica and KV cache block usage, not GPU
  utilisation (which reads near 100% whenever any batch runs).
- *Example* policy: add a replica when queue wait p95 exceeds 1 s for 2
  minutes or block usage exceeds 90%; remove one when block usage stays below
  40% for 15 minutes.
- Cold start (node provisioning, image pull, weight load of 140 GB) can take
  minutes. Keep a minimum of 3 replicas, pre-pull weights to local disk and
  scale ahead of known peaks from the daily pattern.
- GPU capacity may be unavailable when needed; reserve the minimum and the
  expected peak, and use spot capacity only for the batch class.

### 7.7 Quantization trade-offs

| Variant | Weight memory (70B) | KV per token | GPUs per replica | Quality risk |
|---|---|---|---|---|
| BF16 weights and KV | 140 GB | 320 KiB | 4 | Baseline |
| FP8 weights, BF16 KV | 70 GB | 320 KiB | 2 (KV pool about 66 GB) | Low on most tasks; evaluate |
| FP8 weights and FP8 KV | 70 GB | 160 KiB | 2 (about twice the tokens of the row above) | Low to medium; evaluate long context |
| INT4 weight-only | about 35 GB plus scales | 320 KiB | 1 to 2 | Medium; evaluate reasoning and code |

For the FP8 row: $0.9 \times 160 - 70 - 8 = 66$ GB of KV pool on 2 GPUs. Adopt
a variant only after it passes the same evaluation suite as the baseline,
per task, with results recorded in the model change ADR.

### 7.8 Multi-tenancy and rate limits

- Quotas per tenant and model alias in tokens per minute and requests per
  minute; concurrency limit per tenant to stop one tenant filling the batch.
- Priority classes map to queue weights: interactive first, standard next,
  batch only on spare capacity. Batch requests carry long deadlines and are
  preempted first.
- Isolation of data: no cross-tenant prefix sharing unless the prefix is a
  platform-provided system prompt; vector collections are per tenant with
  access checked on every query.
- Load shedding and deadline rules follow the Resilience and Load Shedding
  Design Template (`system-design/resilience-load-shedding-design`).

### 7.9 Vector index

| Index | How it works | Strengths | Weaknesses |
|---|---|---|---|
| Flat (exact) | Compare the query with every vector | Exact recall | Linear cost; small collections only |
| HNSW | Layered proximity graph searched greedily (Malkov and Yashunin) | High recall at low latency; incremental inserts | Memory for graph links; slow rebuilds |
| IVF | Cluster vectors; search only the nearest `nprobe` clusters | Lower memory; fast build | Recall depends on `nprobe` and cluster balance |
| IVF-PQ | IVF plus product-quantized codes | Very small memory | Lower recall; needs re-ranking with full vectors |

Recall versus latency is tuned per collection: for HNSW raise `ef_search`
until recall at 10 reaches the target on a labelled query set, then measure
p99 latency; for IVF raise `nprobe`. Filters applied after search can return
fewer than k results, so use the engine's filtered search or over-fetch.

### 7.10 Embedding pipeline

- Ingest events enter a queue; embedding workers batch texts by length to
  fill GPU batches; results are upserted with document ID and version.
- Each collection pins one embedding model version. Changing the model means
  re-embedding into a new collection and switching an alias, never mixing
  vectors from two models in one index.
- Query embedding runs on a separate low-latency pool so ingest backlogs do
  not slow search.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Interactive latency under mixed load | Priority queues, chunked prefill, per-tenant concurrency caps | KV pool exhaustion at about 244 average sequences per replica |
| Long prompts | Long-context pool, context caps | Pool size |
| Burst from one tenant | Token rate limits and concurrency caps | Gateway counter store throughput |
| Vector search | Shard by collection, replicas for read throughput | Largest single collection in one shard's memory |
| Cold start | Minimum replicas, pre-pulled weights | GPU availability in region |

Latency budget for interactive TTFT (target p95 1.0 s, 1 500-token prompt):

| Step | Budget |
|---|---|
| Gateway auth, quota, routing | 20 ms |
| Queue wait | 500 ms |
| Prefill (3.4) | 130 ms |
| First decode step and streaming | 50 ms |
| Headroom | 300 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| ID | Failure | Impact | Detection | Mitigation |
|---|---|---|---|---|
| FM-01 | GPU or node failure | Streams on that replica cut | Replica health, stream errors | Router stops sending; client retry with idempotent request ID; N+1 capacity |
| FM-02 | KV pool exhaustion | Preemptions, TTFT spike | Block usage, preemption count | Autoscale; lower batch cap; shed batch class |
| FM-03 | Bad model version | Quality regression | Canary evaluation, user feedback | Version pinning; rollback alias |
| FM-04 | Weight store unavailable | Cannot scale up | Pull errors | Node-local cache; regional replica of weights |
| FM-05 | Index shard down | Search fails for a collection | Shard health | Replicas per shard; degrade to keyword search where the application supports it |
| [FM-nn] | [Failure] | [Impact] | [Signal] | [Mitigation] |

### 9.2 Security

| ID | Threat | Control |
|---|---|---|
| SEC-01 | Tenant reads another tenant's vectors | Collection ACL checked on every query; tenant ID in every index key |
| SEC-02 | Prompt and output leakage through logs | Logging off by default; redaction; retention limits |
| SEC-03 | Cache side channel through shared prefixes | Prefix sharing only within a tenant or for platform prompts |
| SEC-04 | Model supply chain (tampered weights) | Signed artifacts, checksum on load, provenance record |
| SEC-05 | Denial of wallet (huge `max_tokens` floods) | Token reservation at admission; per-tenant caps |
| [SEC-nn] | [Threat] | [Control] |

Application-level risks such as prompt injection belong to each tenant's
threat model; see Threat Model Template (`security/threat-model`).

### 9.3 Observability

Instrument with the conventions in LLM and Agent Observability Specification
Template (`ai-data/llm-observability-spec`).

| Signal | Metric | Alert threshold |
|---|---|---|
| TTFT | p95 by pool and priority | Above 1.0 s for 10 min (interactive) |
| TPOT | p95 by pool | Above 50 ms for 10 min |
| Queue wait | p95 by pool and priority | Above 2 s |
| KV block usage | Per replica | Above 95% for 5 min |
| Preemptions | Per minute per replica | Doubling vs. baseline |
| Tokens per second | Prompt and output per pool | Capacity dashboard |
| Recall at 10 | Weekly evaluation per collection | Below 0.95 |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Large pool GPUs | USD 2.50 per GPU-hour (*Example*) | [Average GPUs x 730 h] | [USD] |
| Small pool and embedding GPUs | [USD per GPU-hour] | [GPU-hours] | [USD] |
| Vector index nodes | [USD per node-month] | [Nodes] | [USD] |
| Gateway, router, storage | [USD per month] | [Units] | [USD] |
| **Total** | | | **[USD]** |

Track cost with Cloud Cost Review Template (FinOps) (`ops/finops-cost-review`).

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Fleet model | Per-team GPUs; shared pools | Shared pools with quotas | Utilisation and one place for limits | Noisy-neighbour risk to manage |
| Hosting | External model API; self-hosted | Self-hosted for these models, passthrough for others | Data control and cost at volume | Operations burden and GPU procurement |
| Batching | Static; continuous | Continuous with paged KV cache | Throughput and memory efficiency | Engine dependency |
| Precision | BF16; FP8; INT4 | BF16 at launch, FP8 after evaluation | Quality first, cost later | Twice the GPUs per replica until FP8 |
| Scaling signal | GPU utilisation; queue and KV usage | Queue and KV usage | Reflects latency risk | Engine-specific metrics |
| Vector index | Flat; HNSW; IVF-PQ | HNSW with int8, IVF-PQ for archive | Recall and latency for hot data; memory for cold | Two index types to operate |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Gateway, quotas, small model pool | Two tenants migrated; usage metering reconciled | YYYY-MM-DD |
| M2 | Large model pool with autoscaling | Load test at 1.5 times forecast meets NFR-01 and NFR-02 | YYYY-MM-DD |
| M3 | Retrieval service and HNSW collections | Recall and latency per NFR-05 on each migrated collection | YYYY-MM-DD |
| M4 | Priority classes and batch inference | Mixed-load test meets NFR-04 | YYYY-MM-DD |
| M5 | FP8 variant | Evaluation parity per task; ADR approved | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Tenants move one at a time behind the same API; the old server
stays available for 30 days. Model versions roll out by alias with a 5%
canary and automatic rollback when the canary's evaluation score or error
rate breaches its threshold.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* GPU capacity not available at peak | Medium | High | Reservations for minimum plus peak; second region | [@name] |
| R-02 | *Example:* Measured throughput below estimate | Medium | Medium | Benchmark before purchase; FP8 path | [@name] |
| R-03 | *Example:* Model licence restricts use or scale | Low | High | Licence review per model version | [@name] |
| R-04 | *Example:* Tenants depend on fallback silently | Medium | Medium | Opt-in, header and dashboard | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary, traceability and open questions. -->

### 12.1 Traceability

| Requirement | Design element | Verification |
|---|---|---|
| NFR-01 | 7.3 admission, 7.4 chunked prefill, 7.8 priorities | Load test with prompt length mix |
| NFR-02 | 3.4 decode sizing, batch cap | Load test inter-token intervals |
| NFR-04 | 7.8 priority queues and concurrency caps | Mixed-tenant test |
| NFR-05 | 7.9 index tuning | Recall test set per collection |
| NFR-06 | 3.6 cost model, metering | Monthly chargeback |
| [ID] | [Element] | [Test] |

### 12.2 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we offer a dedicated pool for regulated tenants? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Is disaggregated prefill worth it at our prompt lengths? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Prefill | Processing all prompt tokens to build the KV cache before the first output token |
| Decode | Generating output tokens one step at a time |
| KV cache | Stored attention keys and values for past tokens |
| TTFT | Time to first token, from request receipt to first streamed token |
| TPOT | Time per output token, the mean interval between streamed tokens |
| Continuous batching | Adding and removing sequences from the running batch at each step |
| Recall at k | Share of the true k nearest neighbours that the index returns |
| HNSW | Hierarchical Navigable Small World graph index |
| IVF | Inverted file index over vector clusters |

### 12.4 References

| Reference | Use |
|---|---|
| Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023 (arXiv:2309.06180) | Paged KV cache in 7.3 |
| Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI 2022 | Iteration-level batching in 7.3 |
| Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", EMNLP 2023 (arXiv:2305.13245) | KV heads in 3.3 |
| Malkov and Yashunin, "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs" (arXiv:1603.09320) | HNSW in 7.9 |
| RAG System Design Template (`ai-data/rag-system-design`) | Application layer on this platform |
| Capacity Plan Template (`ops/capacity-plan`) | GPU capacity planning |
| [Inference engine and hardware documentation] | Engine settings and measured rates |

## Standard and licence note

Based on common industry practice for LLM serving and vector search and on
the published research in 12.4. Hardware and throughput figures are
illustrative examples, not benchmarks; measure on your own engine, model and
hardware, and check the current documentation of each component. Original
text written for this collection. Template licence: MIT.