ML Experiment Tracking Report Template

Hypothesis and decision rule, setup, config and seeds, data versions, results vs baseline with variance, paired test, ablations, failure analysis, decision

KI & DatenVollständigReporting informed by the NeurIPS Paper Checklist (2026)experiment tracking report mathAktualisiert 22.10.2026

Im Editor öffnen.md herunterladen

Anleitung: Use it when a training or evaluation experiment finishes and the team must decide whether to adopt the change. Fill the hypothesis, primary metric and decision rule first, before results exist, then the per-seed results table. The main pitfall is reporting the best seed of the candidate against the mean of the baseline.

Vorschau

Experiment Report: [Experiment name]

Report for one machine learning experiment or a small group of related runs: the hypothesis, the setup, the exact configuration and seeds, the data versions, the results against the baseline with their variance, the significance test, the ablations, the failure analysis and the decision taken. It records what the experiment tracker holds (runs, metrics, artefacts) in a form that a reviewer can read and a colleague can reproduce. The reporting items follow the spirit of the NeurIPS Paper Checklist (2026 edition) sections on reproducibility, statistical significance and compute. Replace every [bracketed] value and every row marked Example. Worked examples use a support-ticket classifier trained with and without a new label-noise filter.

Document control

Field	Value
Document ID	[EXP-REP-001]
Experiment ID	[Example: EXP-2026-041]
Version	[0.1]
Status	[Draft / In review / Final]
Owner	[@researcher]
Approvers / reviewers	[@research-lead], [@peer-reviewer]
Last updated	YYYY-MM-DD
Classification	[Internal / Confidential]
Tracker project and run IDs	[Tracker URL]; [run IDs]
Code repository and commit	[repo URL] @ [commit SHA]
Pre-registration date	YYYY-MM-DD (hypothesis and decision rule frozen)
Related documents	[Model training plan], [Evaluation plan], [Datasheet]

Approvals

Role	Name	Decision	Date
Experiment owner	[@name]	[Submitted]	YYYY-MM-DD
Peer reviewer	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Research lead	[@name]	[Decision accepted]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	Pre-registration: hypothesis, metric, decision rule
0.2	YYYY-MM-DD	[@name]	Example: Results for 5 seeds added
1.0	YYYY-MM-DD	[@name]	Example: Decision recorded after peer review

Summary
Hypothesis and decision rule
Setup
Configuration and seeds
Data versions
Results versus baseline
Ablations
Failure analysis
Decision and next steps
Reproducibility record
Glossary and references

1. Summary

Item	Content
Question	Example: Does filtering likely mislabelled training examples improve macro-F1 of the ticket classifier?
Result	Example: Macro-F1 rose from 81.20 to 82.30 (mean of 5 seeds); paired difference +1.10, 95% CI [+0.92, +1.28]
Significance	Example: Paired t-test across seeds, t(4) = 17.4, p < 0.001
Cost	Example: About 35 accelerator-hours for all runs, ablations included
Decision	Example: Adopt the filter in the production training pipeline (see section 9)

2. Hypothesis and decision rule

Item	Content
Hypothesis	Example: Removing training examples whose label disagrees with an out-of-fold ensemble prediction with confidence ≥ 0.9 improves macro-F1 on the clean test set
Null hypothesis	Example: The filter does not change mean macro-F1
Primary metric	Example: Macro-F1 on TEST-CLEAN v3
Secondary metrics	Example: Per-class recall, calibration error, training time
Minimum effect of interest	Example: +0.5 points macro-F1
Significance level	Example: Two-sided, alpha = 0.05
Decision rule	Example: Adopt if the 95% CI of the paired difference lies entirely above +0.5 and no class loses more than 2 points recall

Record any change to this section after results were seen in the revision history, with a reason. Such changes turn the experiment into an exploratory one and the report must say so.

3. Setup

3.1 Task and model

[Describe the task, inputs and outputs, and the model family.]

Example: Multi-class classification of customer support tickets into 14 routing queues. Model: a 125M-parameter encoder fine-tuned with a classification head. Inputs are ticket subject and body truncated to 512 tokens.

3.2 Arms compared

Arm	Description	Only difference from baseline
ARM-A baseline (Example)	Current production training recipe	Not applicable
ARM-B candidate (Example)	Same recipe with label-noise filter applied to TRAIN v7	6.8% of training examples removed by the filter
[ARM-x]	[Description]	[Difference]

Give the baseline the same tuning budget as the candidate. If the candidate had a learning-rate sweep, so did the baseline.

3.3 Evaluation

Set	Purpose	Size	Notes
VAL v7 (Example)	Early stopping and checkpoint choice	4,000 tickets	Same distribution as training
TEST-CLEAN v3 (Example)	Primary metric	3,000 tickets	Labels adjudicated by two senior agents
TEST-RECENT (Example)	Drift check	1,500 tickets from the last 30 days	Not used for the decision

4. Configuration and seeds

4.1 Hyperparameters

Setting	ARM-A	ARM-B
Learning rate (Example)	2e-5	2e-5
Batch size (Example)	32	32
Epochs and early stopping (Example)	Max 5, patience 1 on VAL macro-F1	Same
Optimiser (Example)	AdamW, weight decay 0.01	Same
Max sequence length (Example)	512	512
Filter threshold (Example)	Not applicable	0.9

4.2 Seeds and sources of randomness

Source	Controlled by	Values
Weight initialisation of the head	Seed	Example: 11, 22, 33, 44, 55
Data order and dropout	Same seed	Example: As above
Out-of-fold ensemble used by the filter	Fixed seed 7, 5 folds	Example: One filter result reused by every ARM-B seed
Non-deterministic kernels	[Deterministic mode on / off]	Example: On; 9% slower

Run the same seeds for every arm, so that the comparison can be paired by seed. Henderson et al. (2018) and Bouthillier et al. (2021) show that seed and data-order variance alone can produce differences as large as many published improvements.

4.3 Environment

Item	Value
Hardware	Example: 1 accelerator, 24 GB, per run
Framework and library versions	[Pinned lock file in repo]
Container image	[Image digest]
Wall-clock per run	Example: About 70 minutes
Total compute	Example: Main arms 10 runs x 70 min ≈ 11.7 h; ablations 20 runs x 70 min ≈ 23.3 h; filter ensemble ≈ 0.3 h; total ≈ 35.3 accelerator-hours

5. Data versions

Dataset	Version	Content hash	Rows	Change from previous version
TRAIN (Example)	v7	[sha256]	48,200	Added October tickets
TRAIN filtered (Example)	v7-f0.9	[sha256]	44,922	3,278 rows removed (6.8%)
VAL (Example)	v7	[sha256]	4,000	None
TEST-CLEAN (Example)	v3	[sha256]	3,000	None

Check that no ticket ID appears in more than one split, and record the check. Describe datasets fully in a Datasheet (ai-data/datasheet-for-datasets).

6. Results versus baseline

6.1 Primary metric by seed

Seed	ARM-A macro-F1	ARM-B macro-F1	Difference (B minus A)
11 (Example)	81.2	82.4	+1.2
22 (Example)	80.6	81.8	+1.2
33 (Example)	81.9	82.9	+1.0
44 (Example)	80.9	82.1	+1.2
55 (Example)	81.4	82.3	+0.9
Mean	81.20	82.30	+1.10
Sample standard deviation	0.49	0.41	0.14

6.2 Significance and interval

Because both arms share seeds, analyse the per-seed differences $d_i$ . With $n$ seeds, mean difference $\bar d$ and standard deviation $s_d$ :

t = \frac{\bar d}{s_d / \sqrt{n}}, \qquad \text{CI}_{95} = \bar d \pm t_{0.975,\,n-1} \cdot \frac{s_d}{\sqrt{n}}

Example: $\bar d = 1.10$ , $s_d = 0.141$ , $n = 5$ , so the standard error is $0.141 / \sqrt{5} = 0.063$ , $t = 17.4$ with 4 degrees of freedom, and with $t_{0.975,4} = 2.776$ the 95% interval is $1.10 \pm 0.18 = [0.92, 1.28]$ . The interval lies entirely above the minimum effect of interest (+0.5), so the decision rule in section 2 is met.

Check	Result
Test used	Example: Paired t-test over seeds (variance from training randomness)
Complementary test	Example: Paired bootstrap over test examples for seed 11 (10,000 resamples): difference +1.2, 95% CI [+0.4, +2.0]
Multiple comparisons	Example: One primary metric, so no correction; secondary metrics reported as exploratory
Secondary: worst per-class recall change	Example: Queue "Billing disputes" minus 0.8 points (within the 2-point limit)
Secondary: expected calibration error	Example: 0.041 to 0.036

The seed-level test measures variance from training randomness; the bootstrap over examples measures variance from the finite test set. Report both when they could disagree. Dror et al. (2018) give guidance on choosing tests for NLP metrics.

6.3 Analysis flow

Follow the same path for every experiment so that results are comparable across reports. Stop at the first step that fails and record why.

7. Ablations

ID	Variant	Mean macro-F1 (5 seeds)	Difference vs ARM-A	Reading
ABL-001 (Example)	Filter threshold 0.8 (11.5% removed)	81.90	+0.70	Too aggressive: removes hard but correct examples
ABL-002 (Example)	Filter threshold 0.95 (3.1% removed)	81.75	+0.55	Smaller gain
ABL-003 (Example)	Remove a random 6.8% instead of filtered rows	81.05	minus 0.15	Gain is not explained by less data
ABL-004 (Example)	Relabel flagged rows instead of removing them	82.10	+0.90	Close to ARM-B; needs reviewer time
[ABL-nnn]	[Variant]	[Score]	[Difference]	[Reading]

ABL-003 is the control that matters most here: it rules out the explanation that any reduction in training data would have helped.

8. Failure analysis

8.1 Error categories

ID	Category	ARM-A count	ARM-B count	Example ticket ID
ERR-001 (Example)	Billing vs refund confusion	61	44	[ID]
ERR-002 (Example)	Multi-issue tickets	52	50	[ID]
ERR-003 (Example)	Non-English tickets	38	37	[ID]
ERR-004 (Example)	Very short tickets (under 10 words)	29	31	[ID]
ERR-005 (Example)	Test label probably wrong	12	12	[ID]

Counts come from seed 11 on TEST-CLEAN. The filter mostly fixed ERR-001, where the training data had many inconsistent labels. It did not help multi-issue tickets, which need a multi-label formulation.

8.2 What did not work

Item	Observation	Implication
Example: TEST-RECENT	Gain shrank to +0.4 (exploratory)	New product launched; check for drift before release
Example: Runs with non-deterministic kernels	Spread across seeds doubled	Keep deterministic mode for comparisons
[Item]	[Observation]	[Implication]

9. Decision and next steps

Item	Content
Decision	Example: Adopt (rule in section 2 met)
Scope	Example: Production training recipe from TRAIN v8 onward
Conditions	Example: Re-check TEST-RECENT after the next data refresh
Rejected alternatives	Example: Threshold 0.8 (ABL-001); relabelling (ABL-004) deferred because of reviewer cost

ID	Next step	Owner	Due
NXT-001 (Example)	Add filter to the training pipeline with threshold 0.9	[@ml-engineer]	YYYY-MM-DD
NXT-002 (Example)	Test multi-label formulation for ERR-002	[@researcher]	YYYY-MM-DD
NXT-003 (Example)	Send flagged rows to annotation review	[@data-lead]	YYYY-MM-DD

10. Reproducibility record

Item	Location	Done
Code at a tagged commit	[repo @ tag]	[ ]
Config files for every arm and ablation	[Tracker artefacts]	[ ]
Seeds listed and used for all arms	Section 4.2	[ ]
Data versions with hashes	Section 5	[ ]
Environment lock file and container digest	[Path]	[ ]
Raw per-example predictions saved	[Path]	[ ]
Analysis notebook or script	[Path]	[ ]
Compute used reported	Section 4.3	[ ]
Pre-registration date before first result	Document control	[ ]

For a fuller checklist use the Reproducibility Checklist (science/reproducibility-checklist).

11. Glossary and references

Term	Meaning
Arm	One configuration being compared
Seed	A number that fixes random choices such as initialisation and data order
Paired design	Both arms evaluated with the same seeds or the same test items, so differences can be analysed directly
Confidence interval	A range built by a method that, under its assumptions, covers the true difference in 95% of repeated experiments
Ablation	A variant that removes or changes one component to isolate its effect
Pre-registration	Writing the hypothesis and decision rule down before seeing results

References:

Henderson, P. et al. (2018). Deep Reinforcement Learning that Matters. AAAI 2018. arXiv:1709.06560.
Bouthillier, X. et al. (2021). Accounting for Variance in Machine Learning Benchmarks. MLSys 2021. arXiv:2103.03098.
Dror, R., Baumer, G., Shlomov, S. and Reichart, R. (2018). The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing. ACL 2018, pages 1383 to 1392.
NeurIPS Paper Checklist, 2026 edition (sections on reproducibility, statistical significance and compute resources): https://neurips.cc/public/guides/PaperChecklist
Related templates: Model Training Plan (ai-data/model-training-plan), Fine-Tuning Report (ai-data/fine-tuning-report), LLM Evaluation Plan (ai-data/llm-evaluation-plan), Benchmark Report (ai-data/benchmark-report), ML System Design (ai-data/ml-system-design).

Standard and licence note

Based on common machine learning research practice and on the published methods cited above; reporting items are informed by the NeurIPS Paper Checklist (2026 edition), paraphrased in original wording with no checklist text reproduced. The text, tables and worked example are original and the figures are illustrative. Check the current checklist on the NeurIPS website before using this report for a submission. Template licence: MIT.