RAG System Design: [System name]

Design document for a Retrieval-Augmented Generation (RAG) system: a language model that answers from a curated document corpus and cites its sources. It covers the use case and success metrics, ingestion and chunking, embeddings, the index and hybrid search, the query pipeline, grounding and citations, evaluation, latency and cost budgets, freshness, document-level access control, security, monitoring and rollout. Based on common industry practice. Replace every [bracketed] value and every row marked Example.

Document control

Approvals

Revision history

Contents

1. Use case and success metrics
2. Architecture overview
3. Corpus and ingestion
4. Embeddings
5. Index and retrieval
6. Query pipeline
7. Generation, grounding and citations
8. Access control
9. Security
10. Evaluation
11. Latency and cost budgets
12. Freshness and re-indexing
13. Monitoring and operations
14. Failure modes
15. Rollout plan
16. Decisions, open questions and glossary

1. Use case and success metrics

1.1 Problem and users

[Describe who asks questions, about what, and what they do today without the system.] Example: Support agents (about 400 people) answer customer questions about 2,300 help-centre articles and 180 internal runbooks. Today they search three tools and take a median of 6 minutes to find the right procedure.

1.2 Scope

1.3 Success metrics

2. Architecture overview

The system has an offline ingestion path that turns source documents into indexed chunks and an online query path that retrieves, reranks and generates a cited answer.

3. Corpus and ingestion

3.1 Sources

3.2 Parsing and cleaning

[Describe how each format is converted to text, what is removed and what is kept.] Example: Navigation, cookie banners and footers are stripped. Headings are kept as Markdown so chunks can carry their section path. Tables are converted to Markdown tables and never split across chunks. Scanned PDFs are OCR'd and flagged with ocr=true for lower trust.

3.3 Chunking strategy

The expected number of chunks for a document of $T$ tokens with chunk size $c$ and overlap $o$ is approximately

Example: $T = 3{,}000$, $c = 400$, $o = 60$ gives $\lceil 2940 / 340 \rceil = 9$ chunks.

3.4 Chunk metadata

4. Embeddings

4.1 Storage and cost estimate

Index memory for $N$ chunks of dimension $d$ stored as 32-bit floats, with an ANN graph overhead factor $\alpha$:

Example: $N = 120{,}000$, $d = 1{,}024$, $\alpha = 0.5$ gives $M \approx 120{,}000 \times 1{,}024 \times 4 \times 1.5 \approx 0.74$ GB.

One-off embedding cost for a corpus of $N$ chunks with mean length $\bar{t}$ tokens at price $p$ per million tokens:

Example: $120{,}000 \times 420 / 10^6 \times \text{\textdollar}0.10 \approx \text{\textdollar}5.04$ for a full re-embed; daily incremental cost is under $\text{\textdollar}0.10$.

5. Index and retrieval

5.1 Vector index

5.2 Hybrid search and fusion

Lexical search (BM25) catches exact terms such as error codes and product names that embeddings miss; vector search catches paraphrases. Results are merged with reciprocal rank fusion (RRF):

where $R$ is the set of result lists and $k$ is a constant [Example: 60].

6. Query pipeline

7. Generation, grounding and citations

• Instruction: answer only from the supplied passages; if they do not contain the answer, say so and suggest where to look.
• Citation format: [Example: bracketed passage numbers [2] after each claim, mapped by the API to url#section.]
• Grounding check: [Example: every sentence must carry at least one citation; a lightweight entailment check flags sentences whose cited passage does not support them, and flagged answers show a warning.]
• Abstention: [Example: if the top rerank score is below 0.2, return "I could not find this in the knowledge base" with the three nearest articles as links.]
• Display: [Example: sources shown under the answer with titles and last-updated dates; outdated sources (over 12 months) are labelled.]

8. Access control

9. Security

10. Evaluation

10.1 Evaluation data

10.2 Retrieval metrics

Recall at $k$ for a query $q$ with relevant set $\text{Rel}_q$ and top-$k$ results $\text{Ret}_q@k$, averaged over the query set $Q$:

Mean reciprocal rank, where $\text{rank}_q$ is the position of the first relevant result:

10.3 Generation metrics

10.4 Acceptance thresholds

11. Latency and cost budgets

Cost per query, with input tokens $t_{in}$, output tokens $t_{out}$ and prices $p_{in}$, $p_{out}$ per million tokens, plus retrieval cost $c_{r}$:

Example: $t_{in} = 7{,}000$, $t_{out} = 350$, $p_{in} = \text{\textdollar}3$, $p_{out} = \text{\textdollar}15$, $c_r = \text{\textdollar}0.0004$ gives $C_q \approx \text{\textdollar}0.0267$. At 30,000 queries per day the monthly cost is about $\text{\textdollar}24{,}000$; prompt caching of the system prompt and a 20% answer-cache hit rate are planned to reduce it.

12. Freshness and re-indexing

13. Monitoring and operations

Traces record the rewritten query, retrieved chunk IDs with scores, the final prompt size, model version and latency per step. Prompt and answer text are retained for [30] days under the [logging policy].

14. Failure modes

15. Rollout plan

16. Decisions, open questions and glossary

16.1 Decisions

16.2 Open questions

16.3 Glossary

Standard and licence note

Based on common industry practice for retrieval-augmented generation systems, including standard information retrieval metrics (recall@k, mean reciprocal rank) and reciprocal rank fusion. Original text written for this collection; example numbers and prices are illustrative and must be replaced with your own measurements. Template licence: MIT.