# Inference Cost Model: [Product or feature name]

> Cost and unit economics model for serving a machine learning or
> large language model feature: cost per request and per completed
> task, token volumes in and out, the effect of prompt caching and model
> routing, hosted API pricing compared with self-hosted accelerators,
> utilisation, a sensitivity table and the break-even volume. No formal
> standard governs this document; the token-based formulas follow the
> way hosted model providers price usage, and the self-hosted capacity
> logic follows published serving research (Pope et al. 2022; Kwon et
> al. 2023).
> Replace every `[bracketed]` value and every row marked *Example*.
> All prices in the worked example are invented round numbers, not any
> provider's price list. The example is a customer support assistant.

<!--
Guidance: How to use this template
- Measure before you model: take token counts from real traces or a
  representative test set, not from guesses. Section 2 lists what to
  measure.
- Keep one parameter table (section 3) and derive every other number
  from it, so a price change updates the whole model.
- Express results per completed task as well as per request. Users pay
  for outcomes; retries and multi-turn conversations multiply cost.
- Re-run the model whenever the provider price list, the model version,
  the prompt or the traffic forecast changes.
- Common pitfall: comparing a hosted API price with the hourly price of
  one accelerator, ignoring redundancy, peak load, idle time and the
  people who operate the cluster.
-->

## Document control

<!-- Guidance: Record the price-list date and the traffic forecast version: they expire quickly. -->

| Field | Value |
|---|---|
| Document ID | [COST-MOD-001] |
| Product or feature | [Name] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@ml-platform-lead] |
| Approvers / reviewers | [@product-owner], [@finance-partner], [@infra-lead] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal / Confidential] |
| Price list date(s) | [Provider price page, retrieved YYYY-MM-DD] |
| Traffic forecast | [Forecast name and version] |
| Model versions | [Large model id], [small model id], [self-hosted model id] |
| Related documents | [ML system design], [LLM observability spec], [Capacity plan] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Model owner | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Product owner | [@name] | [Decision] | YYYY-MM-DD |
| Finance partner | [@name] | [Decision] | YYYY-MM-DD |
| Infrastructure lead | [@name] | [Decision] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added routing scenario after classifier test |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved for budget planning |

## Contents

1. [Scope and decisions](#1-scope-and-decisions)
2. [Workload profile](#2-workload-profile)
3. [Parameters](#3-parameters)
4. [Cost per request](#4-cost-per-request)
5. [Caching and routing scenarios](#5-caching-and-routing-scenarios)
6. [Cost per task and monthly cost](#6-cost-per-task-and-monthly-cost)
7. [Self-hosted accelerators versus hosted API](#7-self-hosted-accelerators-versus-hosted-api)
8. [Sensitivity analysis](#8-sensitivity-analysis)
9. [Unit economics and guardrails](#9-unit-economics-and-guardrails)
10. [Assumptions risks and review](#10-assumptions-risks-and-review)
11. [Glossary and references](#11-glossary-and-references)

## 1. Scope and decisions

<!-- Guidance: Say which decisions this model supports and which costs are in or out. Questions: are embeddings, retrieval, guardrail models and logging storage included? Pitfall: modelling only the main model call. -->

| ID | Decision supported | Deadline |
|---|---|---|
| DEC-001 *(Example)* | Whether to add a small-model router in front of the large model | YYYY-MM-DD |
| DEC-002 *(Example)* | At what monthly volume to move the small-model tier to self-hosted accelerators | YYYY-MM-DD |
| DEC-003 *(Example)* | Price floor for the paid plan that includes the assistant | YYYY-MM-DD |

| Cost component | In scope | Notes |
|---|---|---|
| Main model calls | Yes | Section 4 |
| Retries and fallbacks | Yes | Retry factor in section 3 |
| Embeddings and retrieval | [Yes / No] | *Example:* No, under 3% of total; tracked separately |
| Guardrail or moderation calls | [Yes / No] | [Notes] |
| Observability and log storage | [Yes / No] | [Notes] |
| Engineering and operations time | Yes, for self-hosting | Section 7 |

## 2. Workload profile

<!-- Guidance: Describe the traffic as distributions, not single numbers. Take the median and the 90th percentile of tokens per request from traces. Pitfall: using prompt length from the design document instead of what production actually sends. -->

| Measure | Median | 90th percentile | Source |
|---|---|---|---|
| Input tokens per request *(Example)* | 3,000 | 5,200 | 7-day trace sample, 20,000 requests |
| of which reusable prefix (system prompt, tools, policy text) *(Example)* | 2,000 | 2,000 | Prompt specification v4 |
| Output tokens per request *(Example)* | 400 | 900 | Same trace sample |
| Requests per completed task *(Example)* | 1.6 | 3 | Conversation logs |
| Share of requests a small model can handle *(Example)* | 60% | Not applicable | Router pilot on 2,000 labelled requests |

Use the median for the base case and the 90th percentile in the
sensitivity analysis (section 8). If the token distribution has a long
tail, model the tail as a separate request class.

## 3. Parameters

<!-- Guidance: One table, one source of truth. Every derived number in the document must trace back to a row here. Mark each value as measured, quoted or assumed. -->

| ID | Parameter | Symbol | Value | Type |
|---|---|---|---|---|
| PAR-001 *(Example)* | Input tokens per request | $T_{in}$ | 3,000 | Measured |
| PAR-002 *(Example)* | Cacheable prefix tokens | $T_{pre}$ | 2,000 | Measured |
| PAR-003 *(Example)* | Output tokens per request | $T_{out}$ | 400 | Measured |
| PAR-004 *(Example)* | Large model input price | $p_{in}$ | USD 3.00 per million tokens | Assumed for illustration |
| PAR-005 *(Example)* | Large model output price | $p_{out}$ | USD 15.00 per million tokens | Assumed for illustration |
| PAR-006 *(Example)* | Cached input read price | $p_{c}$ | USD 0.30 per million tokens (10% of $p_{in}$) | Assumed for illustration |
| PAR-007 *(Example)* | Small model input and output price | $p'_{in}, p'_{out}$ | USD 0.25 and USD 1.25 per million tokens | Assumed for illustration |
| PAR-008 *(Example)* | Cache hit rate on the prefix | $h$ | 0.8 | Measured in pilot |
| PAR-009 *(Example)* | Share routed to the small model | $r$ | 0.6 | Measured in pilot |
| PAR-010 *(Example)* | Retry and fallback factor | $k$ | 1.03 | Measured |
| PAR-011 *(Example)* | Requests per completed task | $q$ | 1.6 | Measured |
| PAR-012 *(Example)* | Monthly requests (base forecast) | $V$ | 1,000,000 | Forecast |
| PAR-013 *(Example)* | Accelerator price | $p_{acc}$ | USD 2.50 per hour | Assumed for illustration |
| PAR-014 *(Example)* | Sustained requests per second per accelerator at the latency target | $\rho$ | 4 | Load test |
| PAR-015 *(Example)* | Target peak utilisation | $u$ | 0.6 | Policy |
| PAR-016 *(Example)* | Peak-to-average traffic ratio | $\pi$ | 3 | Traffic history |
| PAR-017 *(Example)* | Minimum accelerators for redundancy | $G_{min}$ | 2 | Policy |
| PAR-018 *(Example)* | Operations cost for self-hosting | $O$ | USD 4,000 per month | Estimate of engineer time |

Check each provider's price page on the date in Document control.
Some providers also charge a premium to write a prompt into the cache;
this example ignores that premium for clarity. Add it as a parameter if
your provider charges one.

## 4. Cost per request

<!-- Guidance: Show the formula and one fully worked line. Keep input and output separate: output tokens usually cost several times more and are harder to reduce. -->

Without caching, the cost of one request to a token-priced model is:

$$
c_{\text{req}} = \frac{T_{in} \cdot p_{in} + T_{out} \cdot p_{out}}{10^{6}}
$$

*Example:* $(3{,}000 \times 3.00 + 400 \times 15.00) / 10^{6} = (9{,}000 + 6{,}000) / 10^{6}$
= USD 0.015 per request. Output tokens are 12% of the tokens but 40% of
the cost.

With a prefix cache, a share $h$ of the prefix tokens is billed at the
cached price:

$$
c_{\text{cache}} = \frac{(T_{in} - h \cdot T_{pre}) \cdot p_{in} + h \cdot T_{pre} \cdot p_{c} + T_{out} \cdot p_{out}}{10^{6}}
$$

*Example:* cached tokens $0.8 \times 2{,}000 = 1{,}600$; uncached input
$3{,}000 - 1{,}600 = 1{,}400$. Cost $= (1{,}400 \times 3.00 + 1{,}600 \times 0.30 + 400 \times 15.00) / 10^{6}$
$= (4{,}200 + 480 + 6{,}000) / 10^{6}$ = USD 0.01068 per request.

Small model, no cache: $(3{,}000 \times 0.25 + 400 \times 1.25) / 10^{6} = (750 + 500) / 10^{6}$
= USD 0.00125 per request.

## 5. Caching and routing scenarios

<!-- Guidance: Compare a few named scenarios rather than every combination. Each scenario must state the quality evidence that allows it: routing that lowers quality is not a saving. -->

With routing, a share $r$ of requests goes to the small model and the
rest to the large model with caching; retries multiply the result:

$$
c_{\text{eff}} = k \cdot \left[\, r \cdot c_{\text{small}} + (1 - r) \cdot c_{\text{cache}} \,\right]
$$

| ID | Scenario | Cost per request before retries | With retry factor 1.03 | Saving vs SCN-001 | Quality evidence |
|---|---|---|---|---|---|
| SCN-001 *(Example)* | Large model, no cache | USD 0.01500 | USD 0.01545 | Not applicable | Production baseline |
| SCN-002 *(Example)* | Large model, prefix cache h = 0.8 | USD 0.01068 | USD 0.01100 | 28.8% | Output identical; caching does not change the answer |
| SCN-003 *(Example)* | Routing r = 0.6 plus cache on large tier | USD 0.00502 | USD 0.00517 | 66.5% | Router pilot: task success 91.8% vs 92.3%, within agreed 1-point tolerance |
| [SCN-nnn] | [Scenario] | [Cost] | [Cost] | [%] | [Evidence] |

SCN-003 arithmetic: $0.6 \times 0.00125 + 0.4 \times 0.01068 = 0.00075 + 0.004272 = 0.005022$;
$\times 1.03 = 0.00517$.

Routing quality must be measured with the LLM Evaluation Plan
(`ai-data/llm-evaluation-plan`). A router that sends hard requests to
the small model shifts cost into retries, escalations and lost users.

```mermaid
flowchart LR
  Q[Request] --> RT{Router}
  RT -->|simple, 60%| SM[Small model]
  RT -->|complex, 40%| CC{Prefix in cache?}
  CC -->|hit, 80% of prefix| LM[Large model, cached prefix]
  CC -->|miss| LMF[Large model, full price]
  SM --> V{Valid answer?}
  LM --> V
  LMF --> V
  V -->|no, about 3%| RT
  V -->|yes| OUT[Response]
```

## 6. Cost per task and monthly cost

<!-- Guidance: Convert to the units finance and product use: per completed task, per active user and per month. -->

$$
c_{\text{task}} = q \cdot c_{\text{eff}}, \qquad C_{\text{month}} = V \cdot c_{\text{eff}}
$$

| Scenario | Cost per request | Cost per task (q = 1.6) | Monthly cost at V = 1,000,000 |
|---|---|---|---|
| SCN-001 *(Example)* | USD 0.01545 | USD 0.02472 | USD 15,450 |
| SCN-002 *(Example)* | USD 0.01100 | USD 0.01760 | USD 11,000 |
| SCN-003 *(Example)* | USD 0.00517 | USD 0.00828 | USD 5,173 |

Figures include the 1.03 retry factor. Before retries the monthly
costs are USD 15,000, USD 10,680 and USD 5,022.

## 7. Self-hosted accelerators versus hosted API

<!-- Guidance: Size the fleet for peak traffic and redundancy, not average traffic. Include operations cost. Compare against the tier the self-hosted model would replace, at equal measured quality. Pitfall: using peak benchmark throughput instead of throughput at your latency target. -->

The comparison here is for the small-model tier: an open-weight model of
similar measured quality served on rented accelerators, compared with
the small hosted model at USD 0.00125 per request.

Accelerators needed:

$$
G = \max\left(G_{min},\ \left\lceil \frac{\pi \cdot V / (730 \cdot 3{,}600)}{\rho \cdot u} \right\rceil \right)
$$

Monthly self-hosted cost: $C_{self} = G \cdot 730 \cdot p_{acc} + O$,
with 730 hours in an average month. Average utilisation is
$\bar u = (V / (730 \cdot 3{,}600)) / (G \cdot \rho)$.

| Monthly requests | Average req/s | Peak req/s | Accelerators G | Self-hosted cost | Per request | Hosted small model cost | Average utilisation |
|---|---|---|---|---|---|---|---|
| 1,000,000 *(Example)* | 0.38 | 1.14 | 2 | USD 7,650 | USD 0.00765 | USD 1,250 | 4.8% |
| 10,000,000 *(Example)* | 3.81 | 11.42 | 5 | USD 13,125 | USD 0.00131 | USD 12,500 | 19.0% |
| 14,000,000 *(Example)* | 5.33 | 15.98 | 7 | USD 16,775 | USD 0.00120 | USD 17,500 | 19.0% |
| 50,000,000 *(Example)* | 19.03 | 57.08 | 24 | USD 47,800 | USD 0.00096 | USD 62,500 | 19.8% |

Worked line for 10,000,000: average $10^{7} / 2{,}628{,}000 = 3.81$ req/s;
peak $3.81 \times 3 = 11.42$; capacity per accelerator at target
utilisation $4 \times 0.6 = 2.4$; $11.42 / 2.4 = 4.76$, rounded up to 5;
cost $5 \times 730 \times 2.50 + 4{,}000 = 9{,}125 + 4{,}000$ = USD 13,125.

Reading: in this example self-hosting becomes consistently cheaper from
about 15 million requests per month (14.9 million with these parameters).
Between about 10.5 and 14.9 million the two options alternate, because
each added accelerator is a step of USD 1,825 per month; treat that band
as a tie. Below it, the redundancy minimum and the operations cost
dominate. Average utilisation stays near 20%
because the fleet is sized for a peak three times the average; batch or
off-peak workloads on the same fleet raise utilisation and lower the
break-even point. Serving efficiency depends strongly on batching and
memory management (Kwon et al. 2023) and on the partitioning choices for
large models (Pope et al. 2022), so measure $\rho$ with your own serving
stack. See LLM Inference Serving System Design
(`system-design/design-llm-inference-serving`) for the serving design.

| Factor | Hosted API | Self-hosted |
|---|---|---|
| Cost shape | Variable, per token | Mostly fixed per accelerator-hour plus people |
| Scaling | Provider handles; rate limits apply | You plan capacity and lead times |
| Model choice | Provider's catalogue | Open-weight models you are licensed to run |
| Data handling | Provider's terms and region options | Stays in your environment |
| Operational burden | Low | On-call, upgrades, security patching |

## 8. Sensitivity analysis

<!-- Guidance: Vary one parameter at a time around the base case (SCN-003) and show the effect on cost per request. Highlight the parameters that move cost most; those are where engineering effort pays. -->

Base case: SCN-003, USD 0.00517 per request including retries.

| Parameter varied | Low value | Cost at low | High value | Cost at high |
|---|---|---|---|---|
| Output tokens $T_{out}$ *(Example)* | 200 | USD 0.00378 | 800 | USD 0.00795 |
| Cache hit rate $h$ *(Example)* | 0.0 | USD 0.00695 | 0.95 | USD 0.00484 |
| Routing share $r$ *(Example)* | 0.0 | USD 0.01100 | 0.8 | USD 0.00323 |
| All prices *(Example)* | x 0.5 | USD 0.00259 | x 1.5 | USD 0.00776 |
| Intermediate cache value $h$ = 0.5 *(Example)* | 0.5 | USD 0.00584 | Not applicable | Not applicable |
| Intermediate routing value $r$ = 0.4 *(Example)* | 0.4 | USD 0.00712 | Not applicable | Not applicable |

Reading: routing share and output length move cost most in this
example. Shorter answers (a tighter output format) and a better router
are worth more than further cache tuning once the hit rate is above 0.8.

## 9. Unit economics and guardrails

<!-- Guidance: Relate cost to revenue or value per user, then set alerts that catch cost drift before the invoice does. -->

| Measure | Formula | Value |
|---|---|---|
| Tasks per active user per month *(Example)* | From product analytics | 25 |
| AI cost per active user per month *(Example)* | 25 x USD 0.00828 | USD 0.207 |
| Plan price per user per month *(Example)* | Price list | USD 12.00 |
| AI cost as share of plan price *(Example)* | 0.207 / 12.00 | 1.7% |
| Target ceiling *(Example)* | Policy | 5% of plan price |

| ID | Guardrail | Threshold | Action |
|---|---|---|---|
| GRD-001 *(Example)* | Cost per request, daily average | Above USD 0.0065 (25% over base) | Alert model owner; check output length and cache hit rate |
| GRD-002 *(Example)* | Cache hit rate | Below 0.6 for 1 hour | Alert; check for a prompt change that broke the prefix |
| GRD-003 *(Example)* | Routing share | Outside 0.5 to 0.7 for 1 day | Review router drift |
| GRD-004 *(Example)* | Per-user daily spend | Above 20x median | Rate limit and review for abuse |

Feed these signals from the LLM Observability Specification
(`ai-data/llm-observability-spec`).

## 10. Assumptions risks and review

<!-- Guidance: List the assumptions that would change a decision if wrong, and when the model is reviewed. -->

| ID | Assumption or risk | Effect if wrong | Mitigation |
|---|---|---|---|
| ASM-001 *(Example)* | Provider prices stay as on the price-list date | Costs scale with the price change (section 8) | Review monthly; contract terms |
| ASM-002 *(Example)* | Routing does not reduce task success | Hidden cost through retries and churn | Weekly quality sample on routed traffic |
| ASM-003 *(Example)* | Prompt prefix stays stable | Cache hit rate collapses | Version the prefix; GRD-002 |
| ASM-004 *(Example)* | Load-test throughput holds in production | More accelerators needed | Re-measure after each serving upgrade |
| ASM-005 *(Example)* | Traffic forecast | Break-even point moves | Scenario at 0.5x and 2x forecast |

Review cadence: [*Example:* monthly, and on any model, prompt or price
change].

## 11. Glossary and references

<!-- Guidance: Define pricing and capacity terms so finance and engineering read the numbers the same way. -->

| Term | Meaning |
|---|---|
| Token | The unit providers bill by; a word is often one to two tokens |
| Prefix cache | Reuse of computation for a repeated start of the prompt, billed at a lower rate by some providers |
| Routing | Sending each request to the cheapest model expected to answer it well |
| Utilisation | Share of available serving capacity actually used |
| Break-even volume | Volume at which two options cost the same |
| Retry factor | Average number of model calls per user request, including retries and fallbacks |

References:

- Pope, R. et al. (2022). *Efficiently Scaling Transformer Inference.*
  arXiv:2211.05102.
- Kwon, W. et al. (2023). *Efficient Memory Management for Large
  Language Model Serving with PagedAttention.* SOSP 2023.
  arXiv:2309.06180.
- Kaplan, J. et al. (2020). *Scaling Laws for Neural Language Models.*
  arXiv:2001.08361 (forward pass of about 2N FLOPs per token, useful
  for a first throughput estimate).
- [Provider price pages, with retrieval dates]
- Related templates: ML System Design (`ai-data/ml-system-design`),
  LLM Inference Serving System Design
  (`system-design/design-llm-inference-serving`), Back-of-the-Envelope
  Estimation (`system-design/back-of-envelope-estimation`), Model
  Training Plan (`ai-data/model-training-plan`).

## Standard and licence note

Based on common practice for cost modelling of token-priced model APIs
and self-hosted inference, and on the published serving research cited
above. No formal standard governs this document. The text, formulas as
laid out here and the worked example are original; every price, volume
and throughput figure is invented for illustration and internally
consistent only. Check each provider's current official price page and
your own load-test results before making a decision. Template licence:
MIT.