LLM Inference Cost Model Template

Cost per request and task, tokens in and out, caching and routing scenarios, self-hosted GPU vs API break-even, utilisation, sensitivity table, cost guardrails

AI 및 데이터전체inference cost model math업데이트됨 2026. 10. 22.

에디터에서 열기.md 다운로드

사용법: Use it when you need to budget an LLM feature, choose between hosted APIs and self-hosting, or justify caching and routing work. Fill the workload profile from real traces and the single parameter table first; every other number derives from them. The main pitfall is comparing an API price with one accelerator's hourly price while ignoring peak sizing, redundancy and operations cost.

미리보기

Inference Cost Model: [Product or feature name]

Cost and unit economics model for serving a machine learning or large language model feature: cost per request and per completed task, token volumes in and out, the effect of prompt caching and model routing, hosted API pricing compared with self-hosted accelerators, utilisation, a sensitivity table and the break-even volume. No formal standard governs this document; the token-based formulas follow the way hosted model providers price usage, and the self-hosted capacity logic follows published serving research (Pope et al. 2022; Kwon et al. 2023). Replace every [bracketed] value and every row marked Example. All prices in the worked example are invented round numbers, not any provider's price list. The example is a customer support assistant.

Document control

Field	Value
Document ID	[COST-MOD-001]
Product or feature	[Name]
Version	[0.1]
Status	[Draft / In review / Approved / Superseded]
Owner	[@ml-platform-lead]
Approvers / reviewers	[@product-owner], [@finance-partner], [@infra-lead]
Last updated	YYYY-MM-DD
Classification	[Internal / Confidential]
Price list date(s)	[Provider price page, retrieved YYYY-MM-DD]
Traffic forecast	[Forecast name and version]
Model versions	[Large model id], [small model id], [self-hosted model id]
Related documents	[ML system design], [LLM observability spec], [Capacity plan]

Approvals

Role	Name	Decision	Date
Model owner	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Product owner	[@name]	[Decision]	YYYY-MM-DD
Finance partner	[@name]	[Decision]	YYYY-MM-DD
Infrastructure lead	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft
0.2	YYYY-MM-DD	[@name]	Example: Added routing scenario after classifier test
1.0	YYYY-MM-DD	[@name]	Example: Approved for budget planning

Scope and decisions
Workload profile
Parameters
Cost per request
Caching and routing scenarios
Cost per task and monthly cost
Self-hosted accelerators versus hosted API
Sensitivity analysis
Unit economics and guardrails
Assumptions risks and review
Glossary and references

1. Scope and decisions

ID	Decision supported	Deadline
DEC-001 (Example)	Whether to add a small-model router in front of the large model	YYYY-MM-DD
DEC-002 (Example)	At what monthly volume to move the small-model tier to self-hosted accelerators	YYYY-MM-DD
DEC-003 (Example)	Price floor for the paid plan that includes the assistant	YYYY-MM-DD

Cost component	In scope	Notes
Main model calls	Yes	Section 4
Retries and fallbacks	Yes	Retry factor in section 3
Embeddings and retrieval	[Yes / No]	Example: No, under 3% of total; tracked separately
Guardrail or moderation calls	[Yes / No]	[Notes]
Observability and log storage	[Yes / No]	[Notes]
Engineering and operations time	Yes, for self-hosting	Section 7

2. Workload profile

Measure	Median	90th percentile	Source
Input tokens per request (Example)	3,000	5,200	7-day trace sample, 20,000 requests
of which reusable prefix (system prompt, tools, policy text) (Example)	2,000	2,000	Prompt specification v4
Output tokens per request (Example)	400	900	Same trace sample
Requests per completed task (Example)	1.6	3	Conversation logs
Share of requests a small model can handle (Example)	60%	Not applicable	Router pilot on 2,000 labelled requests

Use the median for the base case and the 90th percentile in the sensitivity analysis (section 8). If the token distribution has a long tail, model the tail as a separate request class.

3. Parameters

ID	Parameter	Symbol	Value	Type
PAR-001 (Example)	Input tokens per request	$T_{in}$	3,000	Measured
PAR-002 (Example)	Cacheable prefix tokens	$T_{pre}$	2,000	Measured
PAR-003 (Example)	Output tokens per request	$T_{out}$	400	Measured
PAR-004 (Example)	Large model input price	$p_{in}$	USD 3.00 per million tokens	Assumed for illustration
PAR-005 (Example)	Large model output price	$p_{out}$	USD 15.00 per million tokens	Assumed for illustration
PAR-006 (Example)	Cached input read price	$p_{c}$	USD 0.30 per million tokens (10% of $p_{in}$ )	Assumed for illustration
PAR-007 (Example)	Small model input and output price	$p'_{in}, p'_{out}$	USD 0.25 and USD 1.25 per million tokens	Assumed for illustration
PAR-008 (Example)	Cache hit rate on the prefix	$h$	0.8	Measured in pilot
PAR-009 (Example)	Share routed to the small model	$r$	0.6	Measured in pilot
PAR-010 (Example)	Retry and fallback factor	$k$	1.03	Measured
PAR-011 (Example)	Requests per completed task	$q$	1.6	Measured
PAR-012 (Example)	Monthly requests (base forecast)	$V$	1,000,000	Forecast
PAR-013 (Example)	Accelerator price	$p_{acc}$	USD 2.50 per hour	Assumed for illustration
PAR-014 (Example)	Sustained requests per second per accelerator at the latency target	$\rho$	4	Load test
PAR-015 (Example)	Target peak utilisation	$u$	0.6	Policy
PAR-016 (Example)	Peak-to-average traffic ratio	$\pi$	3	Traffic history
PAR-017 (Example)	Minimum accelerators for redundancy	$G_{min}$	2	Policy
PAR-018 (Example)	Operations cost for self-hosting	$O$	USD 4,000 per month	Estimate of engineer time

Check each provider's price page on the date in Document control. Some providers also charge a premium to write a prompt into the cache; this example ignores that premium for clarity. Add it as a parameter if your provider charges one.

4. Cost per request

Without caching, the cost of one request to a token-priced model is:

c_{\text{req}} = \frac{T_{in} \cdot p_{in} + T_{out} \cdot p_{out}}{10^{6}}

Example: $(3{,}000 \times 3.00 + 400 \times 15.00) / 10^{6} = (9{,}000 + 6{,}000) / 10^{6}$ = USD 0.015 per request. Output tokens are 12% of the tokens but 40% of the cost.

With a prefix cache, a share $h$ of the prefix tokens is billed at the cached price:

c_{\text{cache}} = \frac{(T_{in} - h \cdot T_{pre}) \cdot p_{in} + h \cdot T_{pre} \cdot p_{c} + T_{out} \cdot p_{out}}{10^{6}}

Example: cached tokens $0.8 \times 2{,}000 = 1{,}600$ ; uncached input $3{,}000 - 1{,}600 = 1{,}400$ . Cost $= (1{,}400 \times 3.00 + 1{,}600 \times 0.30 + 400 \times 15.00) / 10^{6}$ $= (4{,}200 + 480 + 6{,}000) / 10^{6}$ = USD 0.01068 per request.

Small model, no cache: $(3{,}000 \times 0.25 + 400 \times 1.25) / 10^{6} = (750 + 500) / 10^{6}$ = USD 0.00125 per request.

5. Caching and routing scenarios

With routing, a share $r$ of requests goes to the small model and the rest to the large model with caching; retries multiply the result:

c_{\text{eff}} = k \cdot \left[\, r \cdot c_{\text{small}} + (1 - r) \cdot c_{\text{cache}} \,\right]

ID	Scenario	Cost per request before retries	With retry factor 1.03	Saving vs SCN-001	Quality evidence
SCN-001 (Example)	Large model, no cache	USD 0.01500	USD 0.01545	Not applicable	Production baseline
SCN-002 (Example)	Large model, prefix cache h = 0.8	USD 0.01068	USD 0.01100	28.8%	Output identical; caching does not change the answer
SCN-003 (Example)	Routing r = 0.6 plus cache on large tier	USD 0.00502	USD 0.00517	66.5%	Router pilot: task success 91.8% vs 92.3%, within agreed 1-point tolerance
[SCN-nnn]	[Scenario]	[Cost]	[Cost]	[%]	[Evidence]

SCN-003 arithmetic: $0.6 \times 0.00125 + 0.4 \times 0.01068 = 0.00075 + 0.004272 = 0.005022$ ; $\times 1.03 = 0.00517$ .

Routing quality must be measured with the LLM Evaluation Plan (ai-data/llm-evaluation-plan). A router that sends hard requests to the small model shifts cost into retries, escalations and lost users.

6. Cost per task and monthly cost

c_{\text{task}} = q \cdot c_{\text{eff}}, \qquad C_{\text{month}} = V \cdot c_{\text{eff}}

Scenario	Cost per request	Cost per task (q = 1.6)	Monthly cost at V = 1,000,000
SCN-001 (Example)	USD 0.01545	USD 0.02472	USD 15,450
SCN-002 (Example)	USD 0.01100	USD 0.01760	USD 11,000
SCN-003 (Example)	USD 0.00517	USD 0.00828	USD 5,173

Figures include the 1.03 retry factor. Before retries the monthly costs are USD 15,000, USD 10,680 and USD 5,022.

7. Self-hosted accelerators versus hosted API

The comparison here is for the small-model tier: an open-weight model of similar measured quality served on rented accelerators, compared with the small hosted model at USD 0.00125 per request.

Accelerators needed:

G = \max\left(G_{min},\ \left\lceil \frac{\pi \cdot V / (730 \cdot 3{,}600)}{\rho \cdot u} \right\rceil \right)

Monthly self-hosted cost: $C_{self} = G \cdot 730 \cdot p_{acc} + O$ , with 730 hours in an average month. Average utilisation is $\bar u = (V / (730 \cdot 3{,}600)) / (G \cdot \rho)$ .

Monthly requests	Average req/s	Peak req/s	Accelerators G	Self-hosted cost	Per request	Hosted small model cost	Average utilisation
1,000,000 (Example)	0.38	1.14	2	USD 7,650	USD 0.00765	USD 1,250	4.8%
10,000,000 (Example)	3.81	11.42	5	USD 13,125	USD 0.00131	USD 12,500	19.0%
14,000,000 (Example)	5.33	15.98	7	USD 16,775	USD 0.00120	USD 17,500	19.0%
50,000,000 (Example)	19.03	57.08	24	USD 47,800	USD 0.00096	USD 62,500	19.8%

Worked line for 10,000,000: average $10^{7} / 2{,}628{,}000 = 3.81$ req/s; peak $3.81 \times 3 = 11.42$ ; capacity per accelerator at target utilisation $4 \times 0.6 = 2.4$ ; $11.42 / 2.4 = 4.76$ , rounded up to 5; cost $5 \times 730 \times 2.50 + 4{,}000 = 9{,}125 + 4{,}000$ = USD 13,125.

Reading: in this example self-hosting becomes consistently cheaper from about 15 million requests per month (14.9 million with these parameters). Between about 10.5 and 14.9 million the two options alternate, because each added accelerator is a step of USD 1,825 per month; treat that band as a tie. Below it, the redundancy minimum and the operations cost dominate. Average utilisation stays near 20% because the fleet is sized for a peak three times the average; batch or off-peak workloads on the same fleet raise utilisation and lower the break-even point. Serving efficiency depends strongly on batching and memory management (Kwon et al. 2023) and on the partitioning choices for large models (Pope et al. 2022), so measure $\rho$ with your own serving stack. See LLM Inference Serving System Design (system-design/design-llm-inference-serving) for the serving design.

Factor	Hosted API	Self-hosted
Cost shape	Variable, per token	Mostly fixed per accelerator-hour plus people
Scaling	Provider handles; rate limits apply	You plan capacity and lead times
Model choice	Provider's catalogue	Open-weight models you are licensed to run
Data handling	Provider's terms and region options	Stays in your environment
Operational burden	Low	On-call, upgrades, security patching

8. Sensitivity analysis

Base case: SCN-003, USD 0.00517 per request including retries.

Parameter varied	Low value	Cost at low	High value	Cost at high
Output tokens $T_{out}$ (Example)	200	USD 0.00378	800	USD 0.00795
Cache hit rate $h$ (Example)	0.0	USD 0.00695	0.95	USD 0.00484
Routing share $r$ (Example)	0.0	USD 0.01100	0.8	USD 0.00323
All prices (Example)	x 0.5	USD 0.00259	x 1.5	USD 0.00776
Intermediate cache value $h$ = 0.5 (Example)	0.5	USD 0.00584	Not applicable	Not applicable
Intermediate routing value $r$ = 0.4 (Example)	0.4	USD 0.00712	Not applicable	Not applicable

Reading: routing share and output length move cost most in this example. Shorter answers (a tighter output format) and a better router are worth more than further cache tuning once the hit rate is above 0.8.

9. Unit economics and guardrails

Measure	Formula	Value
Tasks per active user per month (Example)	From product analytics	25
AI cost per active user per month (Example)	25 x USD 0.00828	USD 0.207
Plan price per user per month (Example)	Price list	USD 12.00
AI cost as share of plan price (Example)	0.207 / 12.00	1.7%
Target ceiling (Example)	Policy	5% of plan price

ID	Guardrail	Threshold	Action
GRD-001 (Example)	Cost per request, daily average	Above USD 0.0065 (25% over base)	Alert model owner; check output length and cache hit rate
GRD-002 (Example)	Cache hit rate	Below 0.6 for 1 hour	Alert; check for a prompt change that broke the prefix
GRD-003 (Example)	Routing share	Outside 0.5 to 0.7 for 1 day	Review router drift
GRD-004 (Example)	Per-user daily spend	Above 20x median	Rate limit and review for abuse

Feed these signals from the LLM Observability Specification (ai-data/llm-observability-spec).

10. Assumptions risks and review

ID	Assumption or risk	Effect if wrong	Mitigation
ASM-001 (Example)	Provider prices stay as on the price-list date	Costs scale with the price change (section 8)	Review monthly; contract terms
ASM-002 (Example)	Routing does not reduce task success	Hidden cost through retries and churn	Weekly quality sample on routed traffic
ASM-003 (Example)	Prompt prefix stays stable	Cache hit rate collapses	Version the prefix; GRD-002
ASM-004 (Example)	Load-test throughput holds in production	More accelerators needed	Re-measure after each serving upgrade
ASM-005 (Example)	Traffic forecast	Break-even point moves	Scenario at 0.5x and 2x forecast

Review cadence: [Example: monthly, and on any model, prompt or price change].

11. Glossary and references

Term	Meaning
Token	The unit providers bill by; a word is often one to two tokens
Prefix cache	Reuse of computation for a repeated start of the prompt, billed at a lower rate by some providers
Routing	Sending each request to the cheapest model expected to answer it well
Utilisation	Share of available serving capacity actually used
Break-even volume	Volume at which two options cost the same
Retry factor	Average number of model calls per user request, including retries and fallbacks

References:

Pope, R. et al. (2022). Efficiently Scaling Transformer Inference. arXiv:2211.05102.
Kwon, W. et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180.
Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361 (forward pass of about 2N FLOPs per token, useful for a first throughput estimate).
[Provider price pages, with retrieval dates]
Related templates: ML System Design (ai-data/ml-system-design), LLM Inference Serving System Design (system-design/design-llm-inference-serving), Back-of-the-Envelope Estimation (system-design/back-of-envelope-estimation), Model Training Plan (ai-data/model-training-plan).

Standard and licence note

Based on common practice for cost modelling of token-priced model APIs and self-hosted inference, and on the published serving research cited above. No formal standard governs this document. The text, formulas as laid out here and the worked example are original; every price, volume and throughput figure is invented for illustration and internally consistent only. Check each provider's current official price page and your own load-test results before making a decision. Template licence: MIT.