# Benchmark Report: [Benchmark or comparison name]

> Report of a benchmark run comparing models or systems: why the
> benchmark was chosen and whether it measures what matters, the
> contamination checks, the evaluation protocol, the exact prompts and
> decoding settings, repeated runs, confidence intervals, statistical
> tests, the comparison table and the limitations. The reporting
> practice draws on HELM (Liang et al., TMLR 2023) for transparent,
> multi-metric reporting, on Miller (2024) for error bars on evaluation
> results, and on Sainz et al. (2023) for contamination.
> Replace every `[bracketed]` value and every row marked *Example*.
> Worked examples compare two candidate language models on a public
> multiple-choice benchmark and an internal question-answering set.

<!--
Guidance: How to use this template
- Fix the protocol (sections 3 to 5) before running any model, and run
  every model with the same harness, prompts and decoding settings.
- Report uncertainty for every score. A table of point estimates
  without intervals invites readers to over-read small gaps.
- Treat public benchmark scores as weak evidence when contamination is
  possible. Pair them with a private or freshly written set.
- Keep raw per-item outputs. Most disputes about a benchmark result are
  settled by looking at individual items.
- Common pitfall: comparing your own run of model A with a number
  copied from another team's paper for model B, under a different
  prompt format.
-->

## Document control

<!-- Guidance: Record the harness version and the exact model snapshots: a model name without a version or date is not reproducible. -->

| Field | Value |
|---|---|
| Document ID | [BENCH-REP-001] |
| Version | [0.1] |
| Status | [Draft / In review / Final] |
| Owner | [@eval-owner] |
| Approvers / reviewers | [@research-lead], [@peer-reviewer] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal / Public] |
| Models under test | [Model id and snapshot date for each] |
| Evaluation harness and version | [Harness name] [version] @ [commit] |
| Run dates | YYYY-MM-DD to YYYY-MM-DD |
| Related documents | [Evaluation plan], [Model card], [Experiment report] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Evaluation owner | [@name] | [Submitted] | YYYY-MM-DD |
| Peer reviewer | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Research lead | [@name] | [Decision] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | Protocol frozen |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Results and contamination checks added |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Final after peer review |

## Contents

1. [Purpose and summary](#1-purpose-and-summary)
2. [Benchmark choice and validity](#2-benchmark-choice-and-validity)
3. [Contamination checks](#3-contamination-checks)
4. [Evaluation protocol](#4-evaluation-protocol)
5. [Prompts and decoding settings](#5-prompts-and-decoding-settings)
6. [Repeated runs and uncertainty](#6-repeated-runs-and-uncertainty)
7. [Statistical tests](#7-statistical-tests)
8. [Comparison table](#8-comparison-table)
9. [Error analysis](#9-error-analysis)
10. [Limitations](#10-limitations)
11. [Reproducibility package](#11-reproducibility-package)
12. [Glossary and references](#12-glossary-and-references)

## 1. Purpose and summary

<!-- Guidance: Say which decision the benchmark informs, then give the headline result with its interval. Write the summary last. -->

| Item | Content |
|---|---|
| Decision informed | *Example:* Choose between Model A and Model B for the internal research assistant |
| Headline | *Example:* On the internal set (n = 1,000), Model B scores 74.5% vs 71.2% for Model A; paired difference +3.3 points, 95% CI [+0.9, +5.7] |
| Test | *Example:* McNemar test on paired items, chi-square 7.31, p = 0.007 |
| Public benchmark | *Example:* Gap of +1.0 points, interval includes zero; possible contamination for Model A (section 3) |
| Recommendation | *Example:* Prefer Model B, subject to the cost and latency review |

## 2. Benchmark choice and validity

<!-- Guidance: Explain why each benchmark is relevant to the decision. Questions: does the benchmark's task resemble real use? Is its answer key reliable? Is it saturated? Pitfall: choosing benchmarks because competitors report them. -->

| ID | Benchmark | Version | What it measures | Why chosen | Known issues |
|---|---|---|---|---|---|
| BM-001 *(Example)* | [Public multiple-choice knowledge benchmark] | [Version, item count] | Broad factual and reasoning knowledge | Common reference point | Public since [year]; label errors reported in some subsets |
| BM-002 *(Example)* | Internal QA set v2 | 1,000 items, frozen YYYY-MM-DD | Answers to staff questions over internal documents | Matches the target task | Written by 6 staff; may reflect their style |
| [BM-nnn] | [Benchmark] | [Version] | [Construct] | [Reason] | [Issues] |

### 2.1 Validity checks

| Check | Question | Finding |
|---|---|---|
| Construct validity | Does a high score mean the capability we care about? | *Example:* BM-002 yes; BM-001 only partly (multiple choice, no long answers) |
| Answer key quality | What share of items have a wrong or ambiguous key? | *Example:* 100-item audit of BM-002: 2 ambiguous items fixed before freezing |
| Headroom | Are top models near the ceiling? | *Example:* BM-001 top public scores within a few points of the ceiling; small gaps are hard to read |
| Coverage | Which sub-areas are missing? | *Example:* No multi-turn questions; covered by the LLM Evaluation Plan |

## 3. Contamination checks

<!-- Guidance: Check whether test items could have been in the training data of each model. Use more than one method, and report results per benchmark and per model. Pitfall: declaring "no contamination" because one check found nothing. -->

Contamination means that test items, or their answers, appeared in a
model's training data, so the score measures recall instead of the
skill. Sainz et al. (2023) argue that contamination should be measured
for each benchmark and model; Golchin and Surdeanu (2024) describe
detection by prompting a model to complete benchmark items from their
start.

| ID | Method | Applies to | Result for Model A | Result for Model B |
|---|---|---|---|---|
| CON-001 *(Example)* | Release date vs model data cut-off | BM-001, BM-002 | BM-001 public before cut-off | BM-001 public before cut-off |
| CON-002 *(Example)* | Guided completion: give the first half of an item, compare continuation with the original | BM-001, 200 items | 9 near-exact completions (4.5%) | 1 near-exact completion (0.5%) |
| CON-003 *(Example)* | Canary or private set never published | BM-002 | Not published; low risk | Not published; low risk |
| CON-004 *(Example)* | Perturbed variants (reworded items, shuffled options) | BM-001, 200 items | Score drops 6 points on variants | Score drops 1 point on variants |
| CON-005 *(Example)* | n-gram overlap with training data | Only where training data is available | Not available (closed model) | Not available |

Reading: Model A shows signs of exposure to BM-001, so its BM-001 score
is reported but not used for the decision. Give the reason when a
check cannot be performed, as in CON-005.

## 4. Evaluation protocol

<!-- Guidance: Describe exactly how each item is run and scored, so that another team gets the same numbers. Include scoring rules for partial, empty and malformed answers. -->

| Step | Rule |
|---|---|
| Item order | *Example:* Fixed order from the frozen file; answer options shuffled with seed per item for CON-004 only |
| Shots | *Example:* BM-001: 5 fixed examples from its development split; BM-002: zero-shot |
| Context | *Example:* BM-002 gives the model the 3 source passages per item, identical for all models |
| Answer extraction | *Example:* BM-001: first option letter in the output matching a strict pattern; BM-002: free text |
| Scoring | *Example:* BM-001 exact match on letter; BM-002 graded correct or incorrect by two people blind to model identity, disagreements adjudicated |
| Malformed outputs | *Example:* Scored as incorrect and counted separately |
| Timeouts and API errors | *Example:* Retried up to 3 times; persistent failures reported, not dropped |
| Blinding | *Example:* Graders see outputs labelled X and Y, mapping held by the evaluation owner |

```mermaid
flowchart LR
  F[Frozen item file] --> H[Harness with fixed prompts]
  H --> MA[Model A]
  H --> MB[Model B]
  MA --> O[Raw outputs per item]
  MB --> O
  O --> X[Answer extraction]
  X --> G[Automatic scoring or blind human grading]
  G --> S[Per-item score table]
  S --> ST[Intervals and tests]
```

## 5. Prompts and decoding settings

<!-- Guidance: Publish the exact prompt templates and every decoding parameter. Small changes in format can move scores by several points. -->

### 5.1 Prompt templates

```text
[System] You answer questions using only the passages provided.
If the passages do not contain the answer, reply "Not in sources".

[User]
Passages:
{passage_1}
{passage_2}
{passage_3}

Question: {question}
Answer in at most three sentences.
```

Store each template in the repository with a version tag, and record a
hash of the rendered prompt per item in the raw output file.

### 5.2 Decoding settings

| Setting | Model A | Model B | Note |
|---|---|---|---|
| Temperature *(Example)* | 0.7 | 0.7 | Same for both; five sampled runs per item (section 6) |
| Top-p *(Example)* | 1.0 | 1.0 | Provider default disabled |
| Max output tokens *(Example)* | 256 | 256 | |
| Stop sequences *(Example)* | None | None | |
| System prompt support *(Example)* | Yes | Yes | |
| Snapshot or version *(Example)* | [model-a-YYYY-MM-DD] | [model-b-YYYY-MM-DD] | Pinned, not an alias |

If a model has a recommended setting that differs, run it both ways and
report both, rather than giving one model its preferred setting only.

## 6. Repeated runs and uncertainty

<!-- Guidance: Separate two sources of uncertainty: the finite set of items, and sampling randomness in the model. Report both. Pitfall: a single run at temperature above zero. -->

### 6.1 Uncertainty from the item sample

For accuracy $\hat p$ on $n$ independent items, a normal-approximation
95% interval is:

$$
\hat p \pm 1.96 \sqrt{\frac{\hat p (1 - \hat p)}{n}}
$$

Miller (2024) recommends treating items as a sample from a larger
population of possible questions and reporting such standard errors;
use clustered standard errors when items come in groups that share a
passage or a template.

*Example (BM-002, n = 1,000):* Model A: $\hat p = 0.712$, standard error
$\sqrt{0.712 \times 0.288 / 1000} = 0.0143$, 95% CI [68.4%, 74.0%].
Model B: $\hat p = 0.745$, standard error 0.0138, 95% CI [71.8%, 77.2%].

### 6.2 Uncertainty from sampling

| Run | Model A accuracy | Model B accuracy |
|---|---|---|
| Seed 1 *(Example)* | 71.0% | 74.9% |
| Seed 2 *(Example)* | 71.6% | 74.2% |
| Seed 3 *(Example)* | 70.9% | 74.6% |
| Seed 4 *(Example)* | 71.5% | 74.3% |
| Seed 5 *(Example)* | 71.0% | 74.5% |
| **Mean** | **71.2%** | **74.5%** |
| **Standard deviation across runs** | 0.32 points | 0.27 points |

Run-to-run spread is small compared with the item-sample standard
error, so the item sample is the main source of uncertainty here. For
the paired test in section 7, each item is scored by majority vote over
the five runs; in this example the majority-vote accuracy (712 and 745
correct items) happens to equal the run mean, which will not hold in
general.

## 7. Statistical tests

<!-- Guidance: Use a paired test when both models answer the same items. State the test, the statistic, the p-value and the interval for the difference. Correct for multiple comparisons when you test many benchmarks or many pairs. -->

### 7.1 Paired comparison on BM-002

| | Model B correct | Model B incorrect |
|---|---|---|
| **Model A correct** | 654 | 58 |
| **Model A incorrect** | 91 | 197 |

*Example:* Model A correct: 654 + 58 = 712; Model B correct:
654 + 91 = 745; total 1,000. Only the discordant cells
$b = 58$ and $c = 91$ carry information about the difference.

$$
\chi^2_{\text{McNemar}} = \frac{(b - c)^2}{b + c} = \frac{(58 - 91)^2}{149} = \frac{1089}{149} = 7.31
$$

With 1 degree of freedom, p ≈ 0.007 (with continuity correction,
chi-square 6.87, p ≈ 0.009). The paired difference is
$(c - b) / n = 0.033$ with standard error
$\sqrt{(b + c)/n^2 - (c - b)^2/n^3} = 0.0122$, so the 95% CI is
[+0.9, +5.7] points.

### 7.2 Test choices

| Situation | Suggested method |
|---|---|
| Same items, binary score, two models | McNemar test or paired bootstrap |
| Same items, continuous score | Paired bootstrap or permutation test over items |
| Many benchmarks or model pairs | Holm or Benjamini-Hochberg correction; report the method |
| Graded by people | Report inter-annotator agreement (see Human Evaluation Protocol, `ai-data/human-evaluation-protocol`) |
| Sampling at temperature above zero | Average or majority over repeated runs before the item-level test |

Dror et al. (2018) discuss test selection for NLP metrics in detail.

## 8. Comparison table

<!-- Guidance: One row per benchmark, with interval and n. Mark scores excluded from the decision and say why. Add cost and latency when the decision depends on them. -->

| Benchmark | n | Model A | Model B | Difference (B minus A) | Used for decision |
|---|---|---|---|---|---|
| BM-002 internal QA *(Example)* | 1,000 | 71.2% [68.4, 74.0] | 74.5% [71.8, 77.2] | +3.3 [+0.9, +5.7] | Yes |
| BM-001 public, original items *(Example)* | 1,200 | 78.5% [76.2, 80.8] | 79.5% [77.2, 81.8] | +1.0, interval includes zero | No: contamination signs for Model A |
| BM-001 public, perturbed variants *(Example)* | 200 | 72.5% | 78.5% | +6.0, exploratory | No: small n |
| Median latency per item *(Example)* | 1,000 | 1.9 s | 2.4 s | +0.5 s | Yes, with cost review |
| Malformed outputs *(Example)* | 1,000 | 0.6% | 0.2% | minus 0.4 points | Reported |

Never put a number from another source in this table unless it was
produced with the same protocol; if you must show one, put it in a
separate, clearly labelled row.

## 9. Error analysis

<!-- Guidance: Read a sample of failures for each model and group them. Numbers say who won; errors say why, and whether the gap matters for real use. -->

| ID | Error type | Model A count | Model B count | Note |
|---|---|---|---|---|
| BER-001 *(Example)* | Answer not supported by passages | 112 | 86 | Main source of the gap |
| BER-002 *(Example)* | Said "Not in sources" when the answer was present | 61 | 58 | Similar |
| BER-003 *(Example)* | Correct but over three sentences | 47 | 22 | Not scored as wrong; affects usability |
| BER-004 *(Example)* | Wrong answer key found during review | 3 | 3 | Items fixed in v3; v2 results unchanged |

Counts from one run per model (seed 1); BER-001 and BER-002 are subsets
of incorrect items.

## 10. Limitations

<!-- Guidance: State honestly what the result does not show. Readers trust a report more when its limits are explicit. -->

- *Example:* BM-002 has 1,000 items written by six staff members; it
  may not represent questions from other departments.
- *Example:* Human grading was binary; partial answers were scored as
  incorrect, which may understate both models.
- *Example:* Hosted models may change behind a pinned name; results
  hold for the snapshot dates in Document control.
- *Example:* Contamination checks for closed models are indirect and
  can miss exposure.
- *Example:* Latency was measured from one region at one time of day.
- [Limitation]

## 11. Reproducibility package

<!-- Guidance: List everything another team needs to rerun the benchmark and check the numbers. -->

| Item | Location | Done |
|---|---|---|
| Frozen item files with hashes | [Path] | [ ] |
| Harness code at a tagged commit | [repo @ tag] | [ ] |
| Prompt templates with versions | [Path] | [ ] |
| Decoding settings and model snapshots | Section 5 | [ ] |
| Raw outputs per item per run | [Path] | [ ] |
| Grading guidelines and grader agreement | [Path] | [ ] |
| Analysis script for intervals and tests | [Path] | [ ] |

## 12. Glossary and references

<!-- Guidance: Define the statistical and benchmark terms used in the report and cite the sources behind the method. -->

| Term | Meaning |
|---|---|
| Contamination | Test items or answers present in a model's training data |
| Construct validity | Whether a benchmark measures the capability it claims to measure |
| Discordant pair | An item one model gets right and the other gets wrong |
| Saturation | When top scores approach the ceiling and gaps become hard to interpret |
| Paired test | A test that compares two models on the same items |
| Clustered standard error | A standard error that accounts for items that share a source |

References:

- Liang, P. et al. (2023). *Holistic Evaluation of Language Models.*
  Transactions on Machine Learning Research. arXiv:2211.09110.
- Miller, E. (2024). *Adding Error Bars to Evals: A Statistical Approach
  to Language Model Evaluations.* arXiv:2411.00640.
- Sainz, O. et al. (2023). *NLP Evaluation in trouble: On the Need to
  Measure LLM Data Contamination for each Benchmark.* Findings of
  EMNLP 2023.
- Golchin, S. and Surdeanu, M. (2024). *Time Travel in LLMs: Tracing
  Data Contamination in Large Language Models.* ICLR 2024.
  arXiv:2308.08493.
- Dror, R. et al. (2018). *The Hitchhiker's Guide to Testing Statistical
  Significance in Natural Language Processing.* ACL 2018.
- Related templates: LLM Evaluation Plan (`ai-data/llm-evaluation-plan`),
  Agent Evaluation Plan (`ai-data/agent-evaluation-plan`), Experiment
  Report (`ai-data/experiment-tracking-report`), Human Evaluation
  Protocol (`ai-data/human-evaluation-protocol`), Model Card
  (`ai-data/model-card`).

## Standard and licence note

Based on common evaluation practice for machine learning models and on
the published methods cited above (HELM, Miller 2024, Sainz et al.
2023, Golchin and Surdeanu 2024, Dror et al. 2018). No formal standard
governs benchmark reports. The text, tables and worked example are
original; all scores and counts are illustrative and internally
consistent only. Check the current version of each benchmark and paper
before citing them. Template licence: MIT.