Benchmark Report: [Benchmark or comparison name]

Report of a benchmark run comparing models or systems: why the benchmark was chosen and whether it measures what matters, the contamination checks, the evaluation protocol, the exact prompts and decoding settings, repeated runs, confidence intervals, statistical tests, the comparison table and the limitations. The reporting practice draws on HELM (Liang et al., TMLR 2023) for transparent, multi-metric reporting, on Miller (2024) for error bars on evaluation results, and on Sainz et al. (2023) for contamination. Replace every [bracketed] value and every row marked Example. Worked examples compare two candidate language models on a public multiple-choice benchmark and an internal question-answering set.

Document control

Approvals

Revision history

Contents

1. Purpose and summary
2. Benchmark choice and validity
3. Contamination checks
4. Evaluation protocol
5. Prompts and decoding settings
6. Repeated runs and uncertainty
7. Statistical tests
8. Comparison table
9. Error analysis
10. Limitations
11. Reproducibility package
12. Glossary and references

1. Purpose and summary

2. Benchmark choice and validity

2.1 Validity checks

3. Contamination checks

Contamination means that test items, or their answers, appeared in a model's training data, so the score measures recall instead of the skill. Sainz et al. (2023) argue that contamination should be measured for each benchmark and model; Golchin and Surdeanu (2024) describe detection by prompting a model to complete benchmark items from their start.

Reading: Model A shows signs of exposure to BM-001, so its BM-001 score is reported but not used for the decision. Give the reason when a check cannot be performed, as in CON-005.

4. Evaluation protocol

5. Prompts and decoding settings

5.1 Prompt templates

[System] You answer questions using only the passages provided.
If the passages do not contain the answer, reply "Not in sources".

[User]
Passages:
{passage_1}
{passage_2}
{passage_3}

Question: {question}
Answer in at most three sentences.

Store each template in the repository with a version tag, and record a hash of the rendered prompt per item in the raw output file.

5.2 Decoding settings

If a model has a recommended setting that differs, run it both ways and report both, rather than giving one model its preferred setting only.

6. Repeated runs and uncertainty

6.1 Uncertainty from the item sample

For accuracy $\hat p$ on $n$ independent items, a normal-approximation 95% interval is:

Miller (2024) recommends treating items as a sample from a larger population of possible questions and reporting such standard errors; use clustered standard errors when items come in groups that share a passage or a template.

Example (BM-002, n = 1,000): Model A: $\hat p = 0.712$, standard error $\sqrt{0.712 \times 0.288 / 1000} = 0.0143$, 95% CI [68.4%, 74.0%]. Model B: $\hat p = 0.745$, standard error 0.0138, 95% CI [71.8%, 77.2%].

6.2 Uncertainty from sampling

Run-to-run spread is small compared with the item-sample standard error, so the item sample is the main source of uncertainty here. For the paired test in section 7, each item is scored by majority vote over the five runs; in this example the majority-vote accuracy (712 and 745 correct items) happens to equal the run mean, which will not hold in general.

7. Statistical tests

7.1 Paired comparison on BM-002

Example: Model A correct: 654 + 58 = 712; Model B correct: 654 + 91 = 745; total 1,000. Only the discordant cells $b = 58$ and $c = 91$ carry information about the difference.

With 1 degree of freedom, p ≈ 0.007 (with continuity correction, chi-square 6.87, p ≈ 0.009). The paired difference is $(c - b) / n = 0.033$ with standard error $\sqrt{(b + c)/n^2 - (c - b)^2/n^3} = 0.0122$, so the 95% CI is [+0.9, +5.7] points.

7.2 Test choices

Dror et al. (2018) discuss test selection for NLP metrics in detail.

8. Comparison table

Never put a number from another source in this table unless it was produced with the same protocol; if you must show one, put it in a separate, clearly labelled row.

9. Error analysis

Counts from one run per model (seed 1); BER-001 and BER-002 are subsets of incorrect items.

10. Limitations

• Example: BM-002 has 1,000 items written by six staff members; it may not represent questions from other departments.
• Example: Human grading was binary; partial answers were scored as incorrect, which may understate both models.
• Example: Hosted models may change behind a pinned name; results hold for the snapshot dates in Document control.
• Example: Contamination checks for closed models are indirect and can miss exposure.
• Example: Latency was measured from one region at one time of day.
• [Limitation]

11. Reproducibility package

12. Glossary and references

References:

• Liang, P. et al. (2023). Holistic Evaluation of Language Models. Transactions on Machine Learning Research. arXiv:2211.09110.
• Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640.
• Sainz, O. et al. (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Findings of EMNLP 2023.
• Golchin, S. and Surdeanu, M. (2024). Time Travel in LLMs: Tracing Data Contamination in Large Language Models. ICLR 2024. arXiv:2308.08493.
• Dror, R. et al. (2018). The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing. ACL 2018.
• Related templates: LLM Evaluation Plan (ai-data/llm-evaluation-plan), Agent Evaluation Plan (ai-data/agent-evaluation-plan), Experiment Report (ai-data/experiment-tracking-report), Human Evaluation Protocol (ai-data/human-evaluation-protocol), Model Card (ai-data/model-card).

Standard and licence note

Based on common evaluation practice for machine learning models and on the published methods cited above (HELM, Miller 2024, Sainz et al. 2023, Golchin and Surdeanu 2024, Dror et al. 2018). No formal standard governs benchmark reports. The text, tables and worked example are original; all scores and counts are illustrative and internally consistent only. Check the current version of each benchmark and paper before citing them. Template licence: MIT.