System Design: [LLM gateway name]

Reference design for an LLM gateway and model router: one internal API in front of several model providers and self-hosted models. It covers the unified API, provider adapters, keys and quotas per tenant, routing and fallback, exact and semantic caching, PII redaction, guardrail hooks, token streaming, cost metering, telemetry with the OpenTelemetry semantic conventions for generative AI (Development status at the time of writing) and the audit trail. Model serving itself is covered by the LLM Inference Serving and Vector Search Platform Design Template (system-design/design-llm-inference-serving). It follows the 12-section structure of the System Design Document Template (system-design/system-design-document) and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative, not benchmarks or vendor prices.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Forty product teams call three external model providers and two self-hosted model pools directly. Each team stores its own provider keys, several of them in application configuration. Nobody can answer how much the company spent on model calls last week, per team, until the invoices arrive. Two incidents in the last quarter came from customer personal data sent to a provider that is not approved for it. When one provider had a regional outage, every team improvised its own fallback, and two of them silently switched to a cheaper model with lower answer quality.]

1.2 Proposed solution

[Example: A single LLM gateway exposes one OpenAI-compatible chat and embedding API to internal callers. Callers authenticate with a gateway key bound to a tenant and a use case; the gateway holds all provider credentials. For each request it checks quotas in tokens and requests, applies the tenant's data policy (redaction, allowed providers, allowed regions), looks up the exact cache, picks a route from an alias such as chat-large to a concrete provider and model, streams the answer back, and emits one usage record and one trace per call. Guardrail hooks run before and after the model call. Fallback to another provider or model happens only along routes the tenant has approved, and is visible in the response and in metering.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

• [Example: Running model servers and GPUs (see LLM Inference Serving and Vector Search Platform Design Template (system-design/design-llm-inference-serving))]
• [Example: Prompt design, retrieval and agent loops inside applications (see RAG System Design Template (ai-data/rag-system-design))]
• [Example: General API management for non-LLM traffic (see API Gateway System Design Template (system-design/design-api-gateway))]
• [Example: Content policy decisions themselves; the gateway runs the classifiers defined in Guardrails and Safety Filter Specification Template (ai-data/guardrails-spec)]
• [Example: Fine-tuning jobs and batch inference windows longer than 24 hours]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

All values are Example planning inputs.

3.2 Concurrent streams and instances

Little's law gives concurrent streams = arrival rate x duration = 1,000 x 6 s = 6,000 open streams at peak. At 1,000 streams per instance the gateway needs 6 instances at full load. Spread across three zones with four instances each (12 in total), losing one zone leaves 8 instances with 8,000 stream capacity, so peak load runs at 75%. Consequence: the gateway must use non-blocking I/O, and autoscaling must count open streams, not CPU, since a waiting stream costs memory and a socket but almost no CPU.

3.3 Token rates

Peak input is 1,000 x 1,200 = 1.2 million tokens per second; peak output is 1,000 x 300 = 300,000 tokens per second. Consequence: request counts alone cannot protect providers or budgets. Quotas must be in tokens, and the gateway must reserve input + max_tokens at admission and settle the actual count at the end (7.2).

3.4 Counter store load

Each request performs one reservation and one settlement, so the counter store sees about 2,000 atomic operations per second at peak. Consequence: one in-memory counter cluster per region is enough; key counters by tenant and alias so a hot tenant lands on one shard and does not slow others.

3.5 Daily volume, storage and spend

3.6 Summary

4. API design

4.1 Endpoints

4.2 Request example

The caller names an alias, never a provider. Gateway extensions travel in headers so standard client libraries keep working.

POST /v1/chat/completions
Authorization: Bearer [gateway key]
X-Use-Case: support-summary
X-Request-Id: 4f1c2e7a-0b9d-4c1e-9a55-2d7e8f0c6b11
X-Data-Class: personal
Content-Type: application/json

{"model": "chat-large", "stream": true, "max_tokens": 400, "messages": [{"role": "system", "content": "[system prompt]"}, {"role": "user", "content": "[ticket text]"}]}

Response headers report what actually happened:

4.3 Errors

Retries by callers carry the same X-Request-Id. The gateway deduplicates non-streaming requests by this ID for 10 minutes (Example), so a retried call after a network error is not billed twice. Streaming responses are not replayed: a retried stream is a new, separately metered request, so clients retry a stream only when no chunk was received.

5. Data model

5.1 Entities

Configuration is small and audited; usage records are high-volume and append-only. No entity stores prompt content unless the tenant opted in.

5.2 Access patterns

5.3 Storage choices

6. High-level design

7. Detailed design

7.1 Request flow

The reservation happens before any provider call, so a tenant over quota never consumes provider capacity. Settlement uses the provider's reported usage when present and the gateway's own count otherwise; both are kept in the usage record so reconciliation can spot adapters that miscount.

7.2 Deep dive 1: quotas and budgets across tenants

Choice: B, with requests per minute as a secondary guard and a monthly currency budget checked at admission against spend so far.

Reason: spend scales with tokens (3.5), and providers enforce their own token limits per account; mirroring them per tenant stops one tenant from exhausting the shared provider account. Input tokens are counted with the route's tokenizer at admission; when the route is unknown until fallback, the gateway reserves using the largest tokenizer ratio of the eligible routes and settles later.

7.3 Deep dive 2: routing and fallback

Choice: B. Each alias has an ordered fallback chain whose routes all passed the alias's evaluation suite and all satisfy the alias's data policy.

Reason: the 2 incidents in 1.1 came from fallback that changed quality or data handling without approval. Fallback is a product decision, not a networking one.

A request is never retried on another route after the first token reached the caller: the caller would see two different answers spliced together. Retry rules and circuit breakers follow the Resilience and Load Shedding Design Template (system-design/resilience-load-shedding-design).

7.4 Deep dive 3: PII redaction and restoration

Choice: B for routes to external providers; A for self-hosted routes in the approved region; C only for data classes that no route may receive.

Reason: most use cases need names in the answer (a support summary must name the customer), so redaction must be reversible inside the gateway while the provider sees placeholders. The mapping lives only in request memory and is never logged.

Rules:

• Redact before the cache lookup, so cache keys and cached values never contain raw personal data.
• Use typed, numbered placeholders ([PERSON_1], [EMAIL_1]); keep the numbering stable within one request so the model can refer back.
• Restore on the streamed output; buffer partial placeholders that split across chunks before writing to the caller.
• Measure recall on a labelled corpus per language; NFR-06 depends on it.

How real systems do it: Uber's GenAI Gateway (paraphrased)

Uber's engineering blog describes a gateway, written in Go, that gives internal teams one interface to third-party and in-house models and mirrors the OpenAI API so existing client libraries work. Before a request leaves for an external vendor, a redaction step swaps detected personal data for numbered placeholders and keeps the mapping, which the gateway then uses to put the original values back into the vendor's answer. Audit logs support cost attribution, security audits and quality evaluation, and each use case passes a security review against the company's data handling standard before it gets access. Source: Wang and Bansal, "Navigating the LLM Landscape: Uber's Innovation with GenAI Gateway", Uber Engineering blog, July 2024. Read the original for details; this box is a summary.

7.5 Deep dive 4: exact and semantic caching

Choice: B on by default for eligible requests; D passed through and metered as cached tokens; C opt-in per route after an evaluation shows the wrong-answer rate below the route's threshold.

Reason: a semantic hit that answers a different question is a quality incident, not a cost saving. Cache entries are scoped to the tenant and the route version and expire when the route's model version changes.

7.6 Streaming and provider adapters

• The gateway streams to callers with server-sent events (text/event-stream, WHATWG HTML Living Standard section 9.2) in one unified chunk format, and converts each provider's streaming events to it, including tool-call deltas and the final usage chunk.
• Adapters normalise finish reasons, error codes and token counts into the gateway's vocabulary, and keep a test suite of recorded provider responses so a provider API change is caught in CI.
• Client disconnect cancels the provider request and settles the tokens generated so far.

7.7 Guardrail hooks

Hook definitions and thresholds live in the Guardrails and Safety Filter Specification Template (ai-data/guardrails-spec); the gateway only runs them and records the decision.

7.8 Telemetry and audit

Emit one span per model call using the OpenTelemetry semantic conventions for generative AI, which have Development status and are now maintained in the open-telemetry/semantic-conventions-genai repository. Pin the convention version you implement and record it in Document control.

Prompt and output content are not recorded on spans by default. Full instrumentation rules are in the LLM and Agent Observability Specification Template (ai-data/llm-observability-spec).

8. Scalability and performance

Latency budget for gateway overhead (target p95 30 ms):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security

Mapping to the OWASP Top 10 for LLM Applications (2026 edition) and related controls; confirm entry numbering against the current list at genai.owasp.org.

Run a full threat model with the Threat Model Template (security/threat-model).

9.3 Observability

9.4 Cost estimate

Track spend with the Cloud Cost Review Template (FinOps) (ops/finops-cost-review).

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Tenants move one use case at a time by changing their base URL; direct provider access remains for 30 days and is then revoked. Route changes roll out to 5% of an alias's traffic first and roll back automatically on error-rate breach.]

11.3 Risks

12. Appendices

12.1 Traceability

12.2 Open questions

12.3 Glossary

12.4 References

Standard and licence note

Based on the OpenTelemetry semantic conventions for generative AI (Development status, Apache-2.0), the WHATWG HTML Living Standard section on server-sent events, RFC 9457, and the OWASP Top 10 for LLM Applications 2026 by the OWASP GenAI Security Project (CC BY-SA 4.0; OWASP entry names are attributed to OWASP, and any adapted OWASP material you redistribute must carry the same licence under its ShareAlike term). The engineering write-up in 7.4 is paraphrased and cited as inspiration only. The text is original; provider behaviour, prices and convention names change often, so check the current official version of each source before approval. Template licence: MIT.