System Design: [Voice agent name]

Reference design for a real-time voice agent that answers phone and browser calls, understands speech as it arrives, decides when the caller has finished, reasons and calls tools with a language model, and speaks back with streaming synthesis. It covers telephony (SIP) and WebRTC ingress, streaming speech recognition, turn detection and barge-in, the model with tools, streaming text to speech, the end-to-end latency budget, interruption handling, call recording consent and failover. It uses the W3C WebRTC Recommendation (13 March 2025), SIP (RFC 3261) and the delay guidance of ITU-T G.114 as reference points. It follows the 12-section skeleton of the System Design Document template and is pre-filled with a worked design for an inbound customer service line to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative examples, not benchmarks. This is a documentation format, not legal advice.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: The service line receives 50 000 calls a day. Callers wait a median of 11 minutes in peak hours; 40% of calls are delivery status, appointment changes and simple account questions. The current menu system (press 1 for) has a 35% abandonment rate and cannot understand free speech. A pilot voice bot built on batch recognition answered after 3 to 4 seconds of silence and talked over callers who paused to think.]

1.2 Proposed solution

[Example: A streaming voice agent. Calls arrive over a SIP trunk or WebRTC at regional media servers. Audio streams to a streaming recogniser; a turn detector combines voice activity and the partial transcript to decide when the caller has finished. The language model receives the transcript and conversation state, calls tools through the shared support back end, and streams text to a speech synthesiser sentence by sentence. If the caller speaks while the agent is talking, playback stops within 200 ms and the agent listens. The agent discloses that it is an AI at the start, asks for recording consent where required, and transfers to a human with a summary on request or when rules say so. Target voice-to-voice latency is p50 at or below 900 ms.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

• [Example: outbound calling campaigns; in the United States, AI-generated voices in calls fall under the TCPA artificial voice rules per the FCC declaratory ruling of 8 February 2024 (FCC 24-17), which needs its own consent design]
• [Example: voice cloning of real staff]
• [Example: voice biometrics for authentication]
• [Example: video calls]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

All values are Example planning inputs.

3.2 Concurrency

Busy hour: 50 000 x 0.12 = 6 000 calls. Concurrent calls: 6 000 x 4 / 60 = 400.

Consequence: 400 simultaneous streaming recognition sessions and up to 400 synthesis streams. Each call is pinned to one media server for its whole duration, so media servers scale by concurrent calls (for example, 200 per server: 2 servers plus one spare per region), and the speech vendors must confirm concurrent stream quotas, not only minutes.

3.3 Audio bandwidth

A G.711 stream with 20 ms packets carries 160 bytes of audio plus 40 bytes of IP, UDP and RTP headers, 50 times a second: about 80 kbit/s per direction at the IP layer. 400 calls x 2 directions x 80 kbit/s = 64 Mbit/s.

Consequence: bandwidth is trivial; latency is not. Put media servers in the region of the carrier interconnect and the speech services in the same region, because every network hop is paid twice per turn.

3.4 Recording storage

Daily call minutes: 50 000 x 4 = 200 000. At 32 kbit/s (4 KB/s): 200 000 x 60 s x 4 KB = 48 GB per day, about 17.5 TB per year.

Consequence: store recordings in object storage with lifecycle tiering and deletion by retention class; index by call ID only. Retention, not volume, is the design driver.

3.5 Model and speech cost

Turns per day: 50 000 x 16 = 800 000. Input tokens: 800 000 x 4 000 = 3.2 billion; output: 800 000 x 60 = 48 million.

Consequence: LLM input dominates. Keep the system prompt and tool definitions stable for prompt caching, and keep the spoken history compact (summarise older turns).

3.6 Summary

4. API design

4.1 Interfaces

4.2 Event example

Every event carries the call ID, a monotonic sequence number and a media timestamp, so latency can be measured per stage and barge-in can truncate the conversation at the exact point the caller heard.

{"call_id": "[id]", "seq": 412, "type": "barge_in", "media_ts_ms": 73120, "agent_text_spoken": "Your order shipped on Monday and should", "agent_text_unspoken": " arrive by Thursday. Is there anything else?"}

4.3 Error handling

5. Data model

5.1 Entities

Call state lives in memory on the media server and is mirrored to a shared store at each turn so another server can resume after failure.

5.2 Access patterns

5.3 Storage choices

6. High-level design

7. Detailed design

7.1 One turn

Stages overlap wherever possible: synthesis starts on the first sentence while the model is still generating the rest.

7.2 Deep dive 1: end-to-end latency budget

Voice-to-voice latency is the time from the end of the caller's speech to the first agent audio reaching the caller. In a cascade it is the sum of the stages that cannot overlap:

where $t_{in}$ and $t_{out}$ are network and jitter buffer delay on each leg, $t_{eot}$ is the time the turn detector waits after speech stops, $t_{asr}$ the time to the final transcript after the turn ends, $t_{llm}$ the model's time to the first sentence, and $t_{tts}$ the synthesiser's time to first audio. With the Example targets:

For reference, ITU-T G.114 (One-way transmission time) notes that most applications see essentially transparent interactivity when one-way mouth-to-ear delay stays below about 150 ms, and recommends that one-way network delay not exceed 400 ms for general planning. The network legs alone must stay well inside those figures because the agent's processing time comes on top of them.

Choice: B. Reason: tool calls, verification and audit need reliable text; streaming at every stage reaches the target. Revisit C for intents without tools (OQ-01).

Techniques that cut perceived latency without cutting corners:

• Start synthesis on the first complete clause, not the full reply.
• While a tool runs longer than about 700 ms, speak a short, honest progress phrase ("Let me check that order.") rather than silence.
• Speculatively start the model on a stable partial transcript; discard the result if the final transcript differs. A speculative run may draft text but never executes a tool call; tools run only from the run on the final transcript.

7.3 Deep dive 2: turn detection, barge-in and interruption handling

Choice: C, with a minimum and maximum wait (for example 200 ms and 1 500 ms) and DTMF for digit strings. Reason: it targets the two failure modes together; digit strings, read out in groups with pauses between them, are the classic case where callers pause mid-turn.

Barge-in and interruption rules:

7.4 Deep dive 3: failover during a live call

A call is a long-lived stateful session; failover must keep the caller on the line.

Choice: C. Reason: vendor degradation is frequent and partial; server loss is rarer but drops every call on that server.

7.5 Recording consent and AI disclosure

• Notice before recording. The greeting states that the caller is speaking with an AI assistant and, where calls are recorded, that the call is recorded and why. The recorder starts only after the notice and, where consent is required, after the caller agrees.
• US recording law varies. Federal law (18 U.S.C. 2511(2)(d)) allows a party to the call, or someone with one party's prior consent, to record, unless the purpose is criminal or tortious. Several states require the consent of all parties; California is a commonly cited example (Penal Code section 632). Apply the strictest rule of the caller's and the company's locations, or ask for consent always.
• EU and UK. Recording personal data needs a lawful basis and transparent information under data protection law; the AI Act Article 50(1) requires informing people that they are interacting with an AI system unless obvious, from 2 August 2026 (check the consolidated text). See AI Content Transparency Plan Template (ai-data/ai-content-transparency-plan).
• Consent declined. Continue without recording and without storing audio; keep a redacted transcript only if the notice allows it, or transfer to a human line that does not record.
• Redaction. Pause recording or mask audio while the caller gives card numbers or verification codes, typically through DTMF capture that never reaches the recorder or the model.

7.6 How real systems do it

How real systems do it (paraphrased from LiveKit, "Using a transformer to improve end of turn detection", blog post of 20 December 2024). LiveKit described adding a small open-weight language model, about 135 million parameters and small enough to run on a CPU within tens of milliseconds, that reads the recent transcript and estimates whether the speaker has finished their thought. Instead of replacing voice activity detection, its prediction lengthens or shortens the silence timeout, so the agent waits when a sentence sounds unfinished and answers promptly when it sounds complete. The post reported a large drop in unintended interruptions compared with voice activity detection alone, at first for English only; later posts describe multilingual and audio-aware versions. Design lesson: treat end of turn as a prediction that adapts the wait, not as a fixed timer (7.3).

8. Scalability and performance

Latency distribution matters more than the mean. Track per-stage p50, p95 and p99 from 5.2 and alert on the stage that moves, not only the total.

9. Reliability, security and operations

9.1 Failure-mode table

9.2 Security

Agent threats beyond voice are covered by Agentic AI Threat Model Template (ai-data/agentic-threat-model) and guardrails by Guardrails Specification Template (ai-data/guardrails-spec).

9.3 Observability and quality

Trace each call with one span per turn and child spans per stage, per LLM and Agent Observability Specification Template (ai-data/llm-observability-spec).

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Route a percentage of calls by number and intent from the existing menu system to the agent; one configuration change routes all calls back to the menu and human queues. Run shadow transcription on human calls first to build the recognition test set.]

11.3 Risks

12. Appendices

12.1 Traceability

12.2 Open questions

12.3 Glossary

12.4 References

Standard and licence note

Based on the W3C WebRTC Recommendation (13 March 2025), IETF RFC 3261 and RFC 3550, ITU-T Recommendation G.114, 18 U.S.C. 2511, FCC 24-17 and Article 50 of Regulation (EU) 2024/1689, with the public engineering write-up in 7.6 paraphrased, not copied. Legal points are summarised in original words and vary by market; this is a documentation format, not legal advice. All numbers are illustrative examples, not benchmarks. Original text written for this collection. Check the current official version of each specification, recommendation and law before use. Template licence: MIT.