# Experiment Report: [Experiment name]

> Report for one machine learning experiment or a small group of
> related runs: the hypothesis, the setup, the exact configuration and
> seeds, the data versions, the results against the baseline with their
> variance, the significance test, the ablations, the failure analysis
> and the decision taken. It records what the experiment tracker holds
> (runs, metrics, artefacts) in a form that a reviewer can read and a
> colleague can reproduce. The reporting items follow the spirit of the
> NeurIPS Paper Checklist (2026 edition) sections on reproducibility,
> statistical significance and compute.
> Replace every `[bracketed]` value and every row marked *Example*.
> Worked examples use a support-ticket classifier trained with and
> without a new label-noise filter.

<!--
Guidance: How to use this template
- Write the hypothesis, metric and decision rule (sections 1 and 2)
  before you look at results. Paste the pre-registration date.
- Link every number to a run ID in your experiment tracker so that it
  can be traced back to code, config and data.
- Report spread across seeds, not only the best run. A gain smaller
  than the seed-to-seed spread is not a finding.
- Write the failure analysis even when the experiment "worked": it is
  where the next hypothesis comes from.
- Common pitfall: changing two things at once and crediting the gain to
  the one you care about. Use the ablation table (section 7).
-->

## Document control

<!-- Guidance: One report per experiment. Keep the run IDs and code commit here so the report never drifts from the evidence. -->

| Field | Value |
|---|---|
| Document ID | [EXP-REP-001] |
| Experiment ID | [*Example:* EXP-2026-041] |
| Version | [0.1] |
| Status | [Draft / In review / Final] |
| Owner | [@researcher] |
| Approvers / reviewers | [@research-lead], [@peer-reviewer] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal / Confidential] |
| Tracker project and run IDs | [Tracker URL]; [run IDs] |
| Code repository and commit | [repo URL] @ [commit SHA] |
| Pre-registration date | YYYY-MM-DD (hypothesis and decision rule frozen) |
| Related documents | [Model training plan], [Evaluation plan], [Datasheet] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Experiment owner | [@name] | [Submitted] | YYYY-MM-DD |
| Peer reviewer | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Research lead | [@name] | [Decision accepted] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | Pre-registration: hypothesis, metric, decision rule |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Results for 5 seeds added |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Decision recorded after peer review |

## Contents

1. [Summary](#1-summary)
2. [Hypothesis and decision rule](#2-hypothesis-and-decision-rule)
3. [Setup](#3-setup)
4. [Configuration and seeds](#4-configuration-and-seeds)
5. [Data versions](#5-data-versions)
6. [Results versus baseline](#6-results-versus-baseline)
7. [Ablations](#7-ablations)
8. [Failure analysis](#8-failure-analysis)
9. [Decision and next steps](#9-decision-and-next-steps)
10. [Reproducibility record](#10-reproducibility-record)
11. [Glossary and references](#11-glossary-and-references)

## 1. Summary

<!-- Guidance: Five lines a manager can read: what was tried, what happened with the uncertainty, and what was decided. Write it last. -->

| Item | Content |
|---|---|
| Question | *Example:* Does filtering likely mislabelled training examples improve macro-F1 of the ticket classifier? |
| Result | *Example:* Macro-F1 rose from 81.20 to 82.30 (mean of 5 seeds); paired difference +1.10, 95% CI [+0.92, +1.28] |
| Significance | *Example:* Paired t-test across seeds, t(4) = 17.4, p < 0.001 |
| Cost | *Example:* About 35 accelerator-hours for all runs, ablations included |
| Decision | *Example:* Adopt the filter in the production training pipeline (see section 9) |

## 2. Hypothesis and decision rule

<!-- Guidance: State a falsifiable hypothesis, the primary metric, the minimum effect worth acting on, and what you will do for each outcome. Questions to answer: what result would make us drop the idea? Pitfall: picking the metric after seeing which one moved. -->

| Item | Content |
|---|---|
| Hypothesis | *Example:* Removing training examples whose label disagrees with an out-of-fold ensemble prediction with confidence ≥ 0.9 improves macro-F1 on the clean test set |
| Null hypothesis | *Example:* The filter does not change mean macro-F1 |
| Primary metric | *Example:* Macro-F1 on TEST-CLEAN v3 |
| Secondary metrics | *Example:* Per-class recall, calibration error, training time |
| Minimum effect of interest | *Example:* +0.5 points macro-F1 |
| Significance level | *Example:* Two-sided, alpha = 0.05 |
| Decision rule | *Example:* Adopt if the 95% CI of the paired difference lies entirely above +0.5 and no class loses more than 2 points recall |

Record any change to this section after results were seen in the
revision history, with a reason. Such changes turn the experiment into
an exploratory one and the report must say so.

## 3. Setup

<!-- Guidance: Describe the task, the model, the baseline and the evaluation in enough detail that someone outside the team understands what was compared. Pitfall: a baseline that was tuned less than the candidate. -->

### 3.1 Task and model

[Describe the task, inputs and outputs, and the model family.]

*Example:* Multi-class classification of customer support tickets into
14 routing queues. Model: a 125M-parameter encoder fine-tuned with a
classification head. Inputs are ticket subject and body truncated to
512 tokens.

### 3.2 Arms compared

| Arm | Description | Only difference from baseline |
|---|---|---|
| ARM-A baseline *(Example)* | Current production training recipe | Not applicable |
| ARM-B candidate *(Example)* | Same recipe with label-noise filter applied to TRAIN v7 | 6.8% of training examples removed by the filter |
| [ARM-x] | [Description] | [Difference] |

Give the baseline the same tuning budget as the candidate. If the
candidate had a learning-rate sweep, so did the baseline.

### 3.3 Evaluation

| Set | Purpose | Size | Notes |
|---|---|---|---|
| VAL v7 *(Example)* | Early stopping and checkpoint choice | 4,000 tickets | Same distribution as training |
| TEST-CLEAN v3 *(Example)* | Primary metric | 3,000 tickets | Labels adjudicated by two senior agents |
| TEST-RECENT *(Example)* | Drift check | 1,500 tickets from the last 30 days | Not used for the decision |

## 4. Configuration and seeds

<!-- Guidance: List every hyperparameter and every source of randomness. Link the config file in the tracker. Seeds must be chosen before running, not selected after. -->

### 4.1 Hyperparameters

| Setting | ARM-A | ARM-B |
|---|---|---|
| Learning rate *(Example)* | 2e-5 | 2e-5 |
| Batch size *(Example)* | 32 | 32 |
| Epochs and early stopping *(Example)* | Max 5, patience 1 on VAL macro-F1 | Same |
| Optimiser *(Example)* | AdamW, weight decay 0.01 | Same |
| Max sequence length *(Example)* | 512 | 512 |
| Filter threshold *(Example)* | Not applicable | 0.9 |

### 4.2 Seeds and sources of randomness

| Source | Controlled by | Values |
|---|---|---|
| Weight initialisation of the head | Seed | *Example:* 11, 22, 33, 44, 55 |
| Data order and dropout | Same seed | *Example:* As above |
| Out-of-fold ensemble used by the filter | Fixed seed 7, 5 folds | *Example:* One filter result reused by every ARM-B seed |
| Non-deterministic kernels | [Deterministic mode on / off] | *Example:* On; 9% slower |

Run the same seeds for every arm, so that the comparison can be paired
by seed. Henderson et al. (2018) and Bouthillier et al. (2021) show that
seed and data-order variance alone can produce differences as large as
many published improvements.

### 4.3 Environment

| Item | Value |
|---|---|
| Hardware | *Example:* 1 accelerator, 24 GB, per run |
| Framework and library versions | [Pinned lock file in repo] |
| Container image | [Image digest] |
| Wall-clock per run | *Example:* About 70 minutes |
| Total compute | *Example:* Main arms 10 runs x 70 min ≈ 11.7 h; ablations 20 runs x 70 min ≈ 23.3 h; filter ensemble ≈ 0.3 h; total ≈ 35.3 accelerator-hours |

## 5. Data versions

<!-- Guidance: Pin every dataset by version and content hash. If preprocessing changed between arms, it is part of the treatment and must be listed. -->

| Dataset | Version | Content hash | Rows | Change from previous version |
|---|---|---|---|---|
| TRAIN *(Example)* | v7 | [sha256] | 48,200 | Added October tickets |
| TRAIN filtered *(Example)* | v7-f0.9 | [sha256] | 44,922 | 3,278 rows removed (6.8%) |
| VAL *(Example)* | v7 | [sha256] | 4,000 | None |
| TEST-CLEAN *(Example)* | v3 | [sha256] | 3,000 | None |

Check that no ticket ID appears in more than one split, and record the
check. Describe datasets fully in a Datasheet
(`ai-data/datasheet-for-datasets`).

## 6. Results versus baseline

<!-- Guidance: Report every seed, the mean, the standard deviation and a confidence interval for the difference. State the test and why it fits the design. Pitfall: reporting the best seed of the candidate against the mean of the baseline. -->

### 6.1 Primary metric by seed

| Seed | ARM-A macro-F1 | ARM-B macro-F1 | Difference (B minus A) |
|---|---|---|---|
| 11 *(Example)* | 81.2 | 82.4 | +1.2 |
| 22 *(Example)* | 80.6 | 81.8 | +1.2 |
| 33 *(Example)* | 81.9 | 82.9 | +1.0 |
| 44 *(Example)* | 80.9 | 82.1 | +1.2 |
| 55 *(Example)* | 81.4 | 82.3 | +0.9 |
| **Mean** | **81.20** | **82.30** | **+1.10** |
| **Sample standard deviation** | 0.49 | 0.41 | 0.14 |

### 6.2 Significance and interval

Because both arms share seeds, analyse the per-seed differences $d_i$.
With $n$ seeds, mean difference $\bar d$ and standard deviation $s_d$:

$$
t = \frac{\bar d}{s_d / \sqrt{n}}, \qquad \text{CI}_{95} = \bar d \pm t_{0.975,\,n-1} \cdot \frac{s_d}{\sqrt{n}}
$$

*Example:* $\bar d = 1.10$, $s_d = 0.141$, $n = 5$, so the standard
error is $0.141 / \sqrt{5} = 0.063$, $t = 17.4$ with 4 degrees of
freedom, and with $t_{0.975,4} = 2.776$ the 95% interval is
$1.10 \pm 0.18 = [0.92, 1.28]$. The interval lies entirely above the
minimum effect of interest (+0.5), so the decision rule in section 2 is
met.

| Check | Result |
|---|---|
| Test used | *Example:* Paired t-test over seeds (variance from training randomness) |
| Complementary test | *Example:* Paired bootstrap over test examples for seed 11 (10,000 resamples): difference +1.2, 95% CI [+0.4, +2.0] |
| Multiple comparisons | *Example:* One primary metric, so no correction; secondary metrics reported as exploratory |
| Secondary: worst per-class recall change | *Example:* Queue "Billing disputes" minus 0.8 points (within the 2-point limit) |
| Secondary: expected calibration error | *Example:* 0.041 to 0.036 |

The seed-level test measures variance from training randomness; the
bootstrap over examples measures variance from the finite test set.
Report both when they could disagree. Dror et al. (2018) give guidance
on choosing tests for NLP metrics.

### 6.3 Analysis flow

Follow the same path for every experiment so that results are
comparable across reports. Stop at the first step that fails and record
why.

```mermaid
flowchart TD
  P[Pre-registered hypothesis and rule] --> R[Run all arms with shared seeds]
  R --> C{Runs complete and valid?}
  C -->|no| F[Fix and rerun; log the failure]
  C -->|yes| S[Per-seed table and paired differences]
  S --> T[Interval and significance test]
  T --> D{Rule met?}
  D -->|yes| A[Ablations confirm the cause]
  D -->|no| N[Reject or design a follow-up]
  A --> W[Decision in section 9]
```

## 7. Ablations

<!-- Guidance: Remove or vary one component at a time to show which part of the change causes the effect. Use the same seeds. Note the cost of each ablation. -->

| ID | Variant | Mean macro-F1 (5 seeds) | Difference vs ARM-A | Reading |
|---|---|---|---|---|
| ABL-001 *(Example)* | Filter threshold 0.8 (11.5% removed) | 81.90 | +0.70 | Too aggressive: removes hard but correct examples |
| ABL-002 *(Example)* | Filter threshold 0.95 (3.1% removed) | 81.75 | +0.55 | Smaller gain |
| ABL-003 *(Example)* | Remove a random 6.8% instead of filtered rows | 81.05 | minus 0.15 | Gain is not explained by less data |
| ABL-004 *(Example)* | Relabel flagged rows instead of removing them | 82.10 | +0.90 | Close to ARM-B; needs reviewer time |
| [ABL-nnn] | [Variant] | [Score] | [Difference] | [Reading] |

ABL-003 is the control that matters most here: it rules out the
explanation that any reduction in training data would have helped.

## 8. Failure analysis

<!-- Guidance: Look at the errors, not only the averages. Sample errors from both arms, group them, and count. Questions: which errors did the change fix, which did it create, and which remain? -->

### 8.1 Error categories

| ID | Category | ARM-A count | ARM-B count | Example ticket ID |
|---|---|---|---|---|
| ERR-001 *(Example)* | Billing vs refund confusion | 61 | 44 | [ID] |
| ERR-002 *(Example)* | Multi-issue tickets | 52 | 50 | [ID] |
| ERR-003 *(Example)* | Non-English tickets | 38 | 37 | [ID] |
| ERR-004 *(Example)* | Very short tickets (under 10 words) | 29 | 31 | [ID] |
| ERR-005 *(Example)* | Test label probably wrong | 12 | 12 | [ID] |

Counts come from seed 11 on TEST-CLEAN. The filter mostly fixed
ERR-001, where the training data had many inconsistent labels. It did
not help multi-issue tickets, which need a multi-label formulation.

### 8.2 What did not work

| Item | Observation | Implication |
|---|---|---|
| *Example:* TEST-RECENT | Gain shrank to +0.4 (exploratory) | New product launched; check for drift before release |
| *Example:* Runs with non-deterministic kernels | Spread across seeds doubled | Keep deterministic mode for comparisons |
| [Item] | [Observation] | [Implication] |

## 9. Decision and next steps

<!-- Guidance: Apply the pre-registered rule and state the decision plainly: adopt, reject, or run a follow-up. Name owners and dates for each follow-up. -->

| Item | Content |
|---|---|
| Decision | *Example:* Adopt (rule in section 2 met) |
| Scope | *Example:* Production training recipe from TRAIN v8 onward |
| Conditions | *Example:* Re-check TEST-RECENT after the next data refresh |
| Rejected alternatives | *Example:* Threshold 0.8 (ABL-001); relabelling (ABL-004) deferred because of reviewer cost |

| ID | Next step | Owner | Due |
|---|---|---|---|
| NXT-001 *(Example)* | Add filter to the training pipeline with threshold 0.9 | [@ml-engineer] | YYYY-MM-DD |
| NXT-002 *(Example)* | Test multi-label formulation for ERR-002 | [@researcher] | YYYY-MM-DD |
| NXT-003 *(Example)* | Send flagged rows to annotation review | [@data-lead] | YYYY-MM-DD |

## 10. Reproducibility record

<!-- Guidance: A colleague should be able to rerun the experiment from this table alone. Tick only what is true today. -->

| Item | Location | Done |
|---|---|---|
| Code at a tagged commit | [repo @ tag] | [ ] |
| Config files for every arm and ablation | [Tracker artefacts] | [ ] |
| Seeds listed and used for all arms | Section 4.2 | [ ] |
| Data versions with hashes | Section 5 | [ ] |
| Environment lock file and container digest | [Path] | [ ] |
| Raw per-example predictions saved | [Path] | [ ] |
| Analysis notebook or script | [Path] | [ ] |
| Compute used reported | Section 4.3 | [ ] |
| Pre-registration date before first result | Document control | [ ] |

For a fuller checklist use the Reproducibility Checklist
(`science/reproducibility-checklist`).

## 11. Glossary and references

<!-- Guidance: Define statistical terms for readers outside the team and cite the method sources used in the analysis. -->

| Term | Meaning |
|---|---|
| Arm | One configuration being compared |
| Seed | A number that fixes random choices such as initialisation and data order |
| Paired design | Both arms evaluated with the same seeds or the same test items, so differences can be analysed directly |
| Confidence interval | A range built by a method that, under its assumptions, covers the true difference in 95% of repeated experiments |
| Ablation | A variant that removes or changes one component to isolate its effect |
| Pre-registration | Writing the hypothesis and decision rule down before seeing results |

References:

- Henderson, P. et al. (2018). *Deep Reinforcement Learning that
  Matters.* AAAI 2018. arXiv:1709.06560.
- Bouthillier, X. et al. (2021). *Accounting for Variance in Machine
  Learning Benchmarks.* MLSys 2021. arXiv:2103.03098.
- Dror, R., Baumer, G., Shlomov, S. and Reichart, R. (2018). *The
  Hitchhiker's Guide to Testing Statistical Significance in Natural
  Language Processing.* ACL 2018, pages 1383 to 1392.
- NeurIPS Paper Checklist, 2026 edition (sections on reproducibility,
  statistical significance and compute resources):
  https://neurips.cc/public/guides/PaperChecklist
- Related templates: Model Training Plan (`ai-data/model-training-plan`),
  Fine-Tuning Report (`ai-data/fine-tuning-report`), LLM Evaluation
  Plan (`ai-data/llm-evaluation-plan`), Benchmark Report
  (`ai-data/benchmark-report`), ML System Design
  (`ai-data/ml-system-design`).

## Standard and licence note

Based on common machine learning research practice and on the
published methods cited above; reporting items are informed by the
NeurIPS Paper Checklist (2026 edition), paraphrased in original wording
with no checklist text reproduced. The text, tables and worked example
are original and the figures are illustrative. Check the current
checklist on the NeurIPS website before using this report for a
submission. Template licence: MIT.