System Design: [Enterprise AI search name]

Reference design for an enterprise search and answer service that retrieves from many workplace systems and answers with citations, while showing each user only what that user may read in the source system. It covers connectors, document-level and chunk-level permission filtering, permission sync, hybrid retrieval and reranking, citation grounding and answer verification, freshness, multi-tenant isolation and evaluation. Threats are mapped to the OWASP Top 10 for LLM Applications 2026, in particular LLM08:2026 Hidden Context Exposure and LLM09:2026 Vector and Embedding Weaknesses. It follows the 12-section skeleton of the System Design Document template and is pre-filled with a worked design for a multi-tenant SaaS to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative examples, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Employees search eight systems separately and cannot find the current policy, the latest design review or who owns a service. A pilot chatbot built on a single shared index answered well, but during testing it quoted a restricted board document to an intern because the index ignored source permissions. Customers of the SaaS product now ask for the same capability across their own workplace systems, with a contractual promise that no user ever sees content they cannot open in the source.]

1.2 Proposed solution

[Example: Connectors pull content and permissions from each source system into a per-tenant index. Every chunk carries the principals allowed to read it. At query time the service resolves the user's identity and group memberships, retrieves only chunks whose principals intersect that set, fuses lexical and vector results, reranks, and passes the top chunks to a model that must cite them. A verifier checks that every sentence in the answer is supported by a cited chunk before the answer is shown. For confidential sources, the top documents are re-checked against the source system's permission API at query time. Each tenant's data lives in its own index partition with its own encryption key.]

Permission guarantee: [Example: An answer or result is built only from chunks the requesting user could open in the source system at the time of the last permission sync, and for confidential sources at the time of the query.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

• [Example: actions in source systems (creating tickets, sending email); see Customer Support AI Agent System Design (system-design/design-customer-support-agent) for an agent with tools]
• [Example: model training on tenant data]
• [Example: field-level redaction inside a chunk (for example, masking a salary column); chunks are visible or not]
• [Example: legal hold and e-discovery exports]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

All values are Example planning inputs.

3.2 Corpus and index

Chunks: 600 million x 8 = 4.8 billion. Vectors at 768 B each (int8): 4.8 billion x 768 B, about 3.7 TB. HNSW links at 64 per vector and 4 B each add 4.8 billion x 256 B, about 1.2 TB. Lexical index and stored text add roughly the size of the text again.

Consequence: about 5 TB of vector data plus text cannot sit on one node. Shard by tenant: the largest tenants get dedicated shards, the long tail shares pooled shards keyed by tenant. Filtered vector search must work inside a shard, so pick an engine whose filters apply during graph traversal, not after.

3.3 Query load

Daily queries: 1.5 million x 0.3 x 6 = 2.7 million. Over 36 000 s that is 75 queries/s on average and about 225 queries/s at peak. Reranking: 225 x 50 = 11 250 query and passage pairs per second at peak.

Consequence: a cross-encoder reranker over 11 250 pairs/s needs its own GPU pool; cap candidates at 50 and skip reranking for navigational queries (exact title or person look-ups) detected by the query classifier.

3.4 Generation tokens and cost

Input tokens per day: 2.7 million x 6 000 = 16.2 billion; output: 2.7 million x 300 = 810 million. At Example prices of USD 1 per million input and USD 4 per million output: 16 200 + 3 240 = USD 19 440 per day.

Consequence: generation is the dominant cost. Answer only when the query is a question (classifier), show results without generation for navigational queries, and cache answers per (tenant, normalised query, permission set hash, index version) for a short time, invalidating any cached answer that cites a document with a revocation event.

3.5 Permission change rate

Documents with permission changes: 600 million x 0.02 = 12 million per day, about 139/s on average. If principals are copied onto each chunk, that is 12 million x 8 = 96 million chunk updates per day, about 1 100/s.

Consequence: permission updates must be partial updates of a small filterable field, not re-indexing of text and vectors. Group membership changes are handled at query time by expanding the user's groups, so a user joining a group never touches the index.

3.6 Summary

4. API design

4.1 Endpoints

4.2 Request and response example

Identity comes from the validated token; the service resolves groups itself. Every citation carries the chunk ID, source link and the source's last modified time so the user can judge freshness.

POST /v1/answer
Authorization: Bearer [user token]
Content-Type: application/json

{"query": "What is the travel approval limit for directors?", "filters": {"sources": ["wiki", "policies"]}, "stream": true}

{"answer": "Directors can approve travel up to [amount] per trip [1].", "citations": [{"id": 1, "chunk_id": "[chunk id]", "title": "Travel policy", "url": "[source link]", "modified": "YYYY-MM-DD"}], "verification": "supported", "index_version": "[version]"}

4.3 Errors

5. Data model

5.1 Entities

Documents and chunks are derived data that can be rebuilt from the source. Identity mappings and ACLs are security data: they need the same review as an authorisation system.

5.2 Access patterns

5.3 Storage choices

6. High-level design

7. Detailed design

7.1 Query flow

The permission filter applies inside retrieval, before any ranking or generation step sees a chunk; the late check narrows further and never widens.

7.2 Deep dive 1: permission filtering and permission sync

Choice: D. Reason: pre-filtering guarantees that nothing outside the user's permissions enters the candidate set or the model context, and the late check closes the sync-lag window for the few sources where a minute of staleness is unacceptable.

Rules for the permission path:

Chunk-level permissions need care at chunking time: never let one chunk span two sections with different restrictions. Split on permission boundaries first, then on structure.

Large principal sets (users in thousands of groups) make filters slow. Use the engine's terms filter on a keyword field, keep the field out of the returned payload, and precompute a per-user principal list when it exceeds a threshold.

7.3 Deep dive 2: hybrid retrieval and reranking

Choice: C. Retrieve 50 from each retriever within the ACL filter, fuse with RRF, rerank the top 50 with a cross-encoder, keep 10 for generation. Reason: enterprise queries mix identifiers ("PRJ-2231 status") and intent ("how do I expense a conference"); RRF needs no score normalisation between retrievers.

RRF scores a document by summing, over each ranked list, one divided by a constant plus its rank:

The constant 60 is the value used by Cormack, Clarke and Buettcher (SIGIR 2009); tune it on your evaluation set. Add freshness and authority signals (for example, official policy space, recent modification) as features in the reranker, not as hand-edited boosts.

7.4 Deep dive 3: citation grounding and answer verification

Choice: C. Reason: an enterprise answer is acted on (policy, numbers, owners); unsupported sentences are the main quality failure.

Verification steps:

1. The generator receives chunks with short IDs and must cite at least one ID per factual sentence.
2. The verifier rejects any citation ID not in the provided set.
3. For each sentence, a small verifier model judges whether the cited chunks support it (supported, partially supported, unsupported).
4. If any sentence is unsupported, the service removes it when the remaining answer still stands, otherwise returns results with "No verified answer found" and the top documents.
5. Verification outcome is stored with the answer for evaluation.

The generator's instructions tell it to answer only from the chunks, to say when they disagree, and to prefer the most recently modified source while naming both.

7.5 Deep dive 4: multi-tenant isolation

Choice: C, with a separate encryption key per tenant for raw content and backups, and a query builder that adds the tenant filter itself and rejects any query without one. Tenant routing follows the cell pattern in Resilience and Load Shedding Design Template (system-design/resilience-load-shedding-design).

7.6 How real systems do it

How real systems do it (paraphrased from Microsoft Learn, "Security filter pattern" for Azure AI Search, page dated August 2026). The documented pattern stores the user and group identifiers that may read a document in a filterable string collection field that is not returned in results, and applies a filter at query time that matches any of the caller's identifiers using a set-membership function rather than a long chain of equality tests, which the page notes is slower and error-prone with many identifiers. The page also warns that hiding the field from results is not itself a security control: authorisation comes from applying the filter on every query. The same documentation points to built-in document-level access control support for sources where it is available. Design lesson: pre-filter by principals on every query and never rely on field visibility for protection (7.2).

8. Scalability and performance

Answer latency budget (target first token p95 2.5 s, Example):

Verification needs complete sentences: buffer the generator stream, release each sentence to the user only after it passes, or show the answer after verification when the interface does not stream. The first-token target therefore applies to the first verified sentence, and the verifier's time for that sentence comes out of the headroom.

9. Reliability, security and operations

9.1 Failure-mode table

9.2 Security

Threat model per Threat Model Template (security/threat-model); privacy per Privacy Threat Model Template (security/privacy-threat-model).

Leakage testing: each tenant environment used for testing holds canary documents restricted to a canary group, with unique marker strings. A daily job queries as users outside that group and fails the build if any marker appears in results, answers or citations. Run the same test after every permission-related change.

9.3 Evaluation

Follow LLM Evaluation Plan Template (ai-data/llm-evaluation-plan).

Labelled sets are built per tenant only with tenant consent; a synthetic cross-industry set covers releases.

9.4 Observability

Trace each query with retrieval, rerank, late check, generation and verification spans per LLM and Agent Observability Specification Template (ai-data/llm-observability-spec). Log chunk IDs, not chunk text.

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Enable per tenant and per connector. Answers are off by default for a tenant until its leakage tests pass. Ranking and model changes roll out behind an index version alias; rollback switches the alias.]

11.3 Risks

R-04 deserves attention: search does not create oversharing, but it makes existing oversharing easy to find. Offer tenants a pre-launch report.

12. Appendices

12.1 Traceability

12.2 Open questions

12.3 Glossary

12.4 References

Standard and licence note

Based on the OWASP Top 10 for LLM Applications 2026 by the OWASP GenAI Security Project (https://genai.owasp.org/), licensed under Creative Commons Attribution-ShareAlike 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), and on the published research and public documentation in 12.4. The OWASP IDs and short names are used for the mapping in 9.2; all descriptions and controls are original text. Because that mapping adapts the OWASP structure, the adapted structure is shared under CC BY-SA 4.0 and any further adaptation must carry the same licence. The box in 7.6 paraphrases public documentation and copies no text. All numbers are illustrative examples, not benchmarks. Check the current official versions of each source before use. Apart from the adapted OWASP structure, the original text is under the template licence. Template licence: MIT.