System Design: [LLM inference platform name]

Reference design for a shared platform that serves large language models and vector search to internal product teams. It covers the request flow (gateway, router, model servers), continuous batching, the KV cache and paged attention, prefill versus decode, GPU memory sizing, latency SLOs (time to first token and time per output token), GPU autoscaling, model routing and fallback, quantization, multi-tenancy and rate limits, the vector index (HNSW and IVF), the embedding pipeline and cost per million tokens. It is the platform underneath applications such as the one in the RAG System Design Template (ai-data/rag-system-design). It follows the structure of the System Design Document template and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative examples, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Six product teams each run their own model server on dedicated GPUs. Average GPU utilisation is 18%, yet two teams hit out-of-memory errors at peak because long prompts exhaust the KV cache. There is no shared rate limiting, so one batch job raised time to first token for the support assistant from 0.6 s to 9 s. Each team also runs its own vector database, with recall nobody has measured.]

1.2 Proposed solution

[Example: A single platform behind an OpenAI-compatible gateway. The gateway authenticates tenants, applies token-based rate limits and passes requests to a router that picks a model pool by model name, prompt length, priority class and pool health. Each pool runs replicas of an inference engine with continuous batching and a paged KV cache (in the style of vLLM PagedAttention). Pools autoscale on queue depth and KV cache usage, not CPU. A separate retrieval service offers embedding and vector search over HNSW indexes per collection. Usage is metered per tenant in tokens and converted to cost per million tokens for chargeback.]

1.3 Decisions requested from reviewers

2. Requirements

2.1 Functional requirements

2.2 Non-functional requirements

2.3 Out of scope

• [Example: model training and fine-tuning pipelines]
• [Example: application-level retrieval logic, prompt design and evaluation (see RAG System Design Template (ai-data/rag-system-design) and LLM Evaluation Plan Template (ai-data/llm-evaluation-plan))]
• [Example: calls to external hosted model APIs (handled by the gateway as a passthrough, not served here)]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

All values are Example planning inputs for the large model pool. Replace them with measurements from your engine and hardware.

3.2 Weights

The weights do not fit one 80 GB GPU, so one replica spans $g = 4$ GPUs (320 GB in total).

3.3 KV cache per token and concurrency

Each layer stores one key and one value vector per KV head per token:

The engine reserves 90% of GPU memory and keeps 8 GB for activations and runtime buffers, leaving for the KV cache pool:

During decode a sequence holds its prompt plus, on average, half its output, so the mean resident length is $1\,500 + 250 = 1\,750$ tokens. A full batch of 128 sequences uses

which fits the 140 GB pool with room for long prompts. The pool could hold about $427\,000 / 1\,750 \approx 244$ average sequences, so with $B = 128$ the batch limit, not memory, binds for average traffic. Long prompts change that: 128 sequences at the 8 192-token maximum would need about 344 GB, so the scheduler must queue or preempt (7.3).

3.4 Decode and prefill rates per replica

Decode reads all weights plus the batch's KV cache once per step:

So TPOT is about 27 ms (within NFR-02) and decode throughput per replica is

Prefill costs about $2P$ floating-point operations per prompt token:

A 1 500-token prompt alone takes about $1\,500 / 11\,400 \approx 0.13$ s to prefill, so the 1 s TTFT target leaves most of the budget for queueing and batching delay.

3.5 Replicas for peak load

Peak demand is $R \times n_{in} = 30\,000$ prompt tokens/s and $R \times n_{out} = 10\,000$ output tokens/s. Treating prefill and decode time as shares of one replica:

At a target utilisation of 0.7 to protect latency, $4.71 / 0.7 \approx 6.7$, rounded up to 7 replicas, plus one for failure and rollout (N+1): 8 replicas, or 32 GPUs at peak.

3.6 Cost per million tokens

At peak the pool costs $32 \times 2.50 = 80$ USD per hour and serves $30\,000 \times 3\,600 = 108$ million prompt tokens and $10\,000 \times 3\,600 = 36$ million output tokens per hour. Splitting cost by the replica time each phase uses (2.63 and 2.08 of 4.71, so 56% and 44%):

These are peak-hour figures. Off-peak capacity that cannot be scaled down raises the daily average; publish the monthly figure from metering, not this estimate.

3.7 Vector index memory

3.8 Summary

4. API design

4.1 Endpoints

4.2 Request example

The tenant names a model alias, not a pool. The router maps the alias to a pinned version and pool; the response reports the version actually used.

POST /v1/chat/completions
Authorization: Bearer [tenant key]
X-Priority: interactive
Content-Type: application/json

{"model": "assistant-large", "stream": true, "max_tokens": 600, "messages": [{"role": "system", "content": "[system prompt]"}, {"role": "user", "content": "[question]"}]}

4.3 Limits and errors

Rate limits count both prompt and output tokens. The gateway reserves prompt_tokens + max_tokens at admission and refunds the unused part on completion, so large max_tokens values cost quota until the stream ends.

5. Data model

5.1 Entities

Configuration entities (tenants, quotas, models, pools) are small and audited; usage records and vectors are high-volume and partitioned.

5.2 Storage choices

6. High-level design

7. Detailed design

7.1 Request flow

One streaming chat request, from admission to metering. The quota reservation happens before queueing so a request that will exceed the tenant limit never occupies GPU memory.

7.2 Model servers and parallelism

• Tensor parallelism splits each layer across the GPUs of one node; it needs fast GPU-to-GPU links, so a replica stays inside one node.
• Pipeline parallelism splits layers across nodes; use it only for models that do not fit one node, because it adds latency per step.
• Data parallelism is the replica count; it is what the autoscaler changes.
• Example: the large pool runs 4-way tensor parallel per replica; the small model runs one replica per GPU.

7.3 Continuous batching, KV cache and paged attention

Static batching waits for every sequence in a batch to finish, so short answers wait for long ones and GPUs idle. Continuous (iteration-level) batching, described for the Orca serving system (Yu et al., OSDI 2022), adds and removes sequences at every decode step, so the batch stays full.

The KV cache stores keys and values of every past token so each new token attends to them without recomputation. Reserving contiguous memory for each sequence's maximum length wastes most of it. PagedAttention (Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023) stores the KV cache in fixed-size blocks mapped through a per-sequence block table, like virtual memory pages. Blocks are allocated as a sequence grows, freed when it ends and can be shared between sequences with a common prefix.

Design consequences:

7.4 Prefill versus decode

Mitigations: split long prompts into chunks interleaved with decode steps (chunked prefill) so decode latency stays bounded; cap prompt tokens prefilled per step; for very long-context tenants consider a separate pool, or disaggregated prefill and decode replicas once the engine supports it.

7.5 Model routing and fallback

Fallback to a different model changes answer quality, so it is opt-in per tenant and always visible in the response and in metering.

7.6 Autoscaling on GPUs

• Scale on pending requests per replica and KV cache block usage, not GPU utilisation (which reads near 100% whenever any batch runs).
• Example policy: add a replica when queue wait p95 exceeds 1 s for 2 minutes or block usage exceeds 90%; remove one when block usage stays below 40% for 15 minutes.
• Cold start (node provisioning, image pull, weight load of 140 GB) can take minutes. Keep a minimum of 3 replicas, pre-pull weights to local disk and scale ahead of known peaks from the daily pattern.
• GPU capacity may be unavailable when needed; reserve the minimum and the expected peak, and use spot capacity only for the batch class.

7.7 Quantization trade-offs

For the FP8 row: $0.9 \times 160 - 70 - 8 = 66$ GB of KV pool on 2 GPUs. Adopt a variant only after it passes the same evaluation suite as the baseline, per task, with results recorded in the model change ADR.

7.8 Multi-tenancy and rate limits

• Quotas per tenant and model alias in tokens per minute and requests per minute; concurrency limit per tenant to stop one tenant filling the batch.
• Priority classes map to queue weights: interactive first, standard next, batch only on spare capacity. Batch requests carry long deadlines and are preempted first.
• Isolation of data: no cross-tenant prefix sharing unless the prefix is a platform-provided system prompt; vector collections are per tenant with access checked on every query.
• Load shedding and deadline rules follow the Resilience and Load Shedding Design Template (system-design/resilience-load-shedding-design).

7.9 Vector index

Recall versus latency is tuned per collection: for HNSW raise ef_search until recall at 10 reaches the target on a labelled query set, then measure p99 latency; for IVF raise nprobe. Filters applied after search can return fewer than k results, so use the engine's filtered search or over-fetch.

7.10 Embedding pipeline

• Ingest events enter a queue; embedding workers batch texts by length to fill GPU batches; results are upserted with document ID and version.
• Each collection pins one embedding model version. Changing the model means re-embedding into a new collection and switching an alias, never mixing vectors from two models in one index.
• Query embedding runs on a separate low-latency pool so ingest backlogs do not slow search.

8. Scalability and performance

Latency budget for interactive TTFT (target p95 1.0 s, 1 500-token prompt):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security

Application-level risks such as prompt injection belong to each tenant's threat model; see Threat Model Template (security/threat-model).

9.3 Observability

Instrument with the conventions in LLM and Agent Observability Specification Template (ai-data/llm-observability-spec).

9.4 Cost estimate

Track cost with Cloud Cost Review Template (FinOps) (ops/finops-cost-review).

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Tenants move one at a time behind the same API; the old server stays available for 30 days. Model versions roll out by alias with a 5% canary and automatic rollback when the canary's evaluation score or error rate breaches its threshold.]

11.3 Risks

12. Appendices

12.1 Traceability

12.2 Open questions

12.3 Glossary

12.4 References

Standard and licence note

Based on common industry practice for LLM serving and vector search and on the published research in 12.4. Hardware and throughput figures are illustrative examples, not benchmarks; measure on your own engine, model and hardware, and check the current documentation of each component. Original text written for this collection. Template licence: MIT.