# ML Reproducibility and Validity Checklist: [Study or paper title]

> Reproducibility and validity checklist for research that uses machine
> learning to make a scientific claim. Structure aligned with REFORMS
> (Kapoor et al., Science Advances, 2024): 32 items in 8 modules, here
> paraphrased in original words, with answer and evidence fields, plus
> working sections on data leakage, evaluation, code and data release,
> and compute reporting. Replace every `[bracketed]` value and every row
> marked *Example*.

<!--
Guidance: How to use this template
- Use it in three moments: when designing the study (to avoid leakage and
  weak baselines), before submission (to check reporting), and as a
  supplement released with the paper. REFORMS suggests the completed
  checklist can be shared alongside the paper as supplementary material.
- Every answer needs evidence: a section, figure, table, file path or
  commit. "Yes" without a pointer counts as "No".
- Modules 1 to 8 follow REFORMS. Sections 9 to 12 are working tools that
  help you answer the leakage, evaluation, release and compute items well.
- This checklist is about ML-based science: claims about the world drawn
  from models. For general code-and-data reproducibility in any research,
  see Reproducibility Checklist (`science/reproducibility-checklist`); for
  the paper itself and a venue checklist, see ML Conference Paper
  (`academic/ml-conference-paper`).
- Common pitfall: treating reproducibility (same code, same numbers) as
  validity. A leaky pipeline reproduces perfectly and is still wrong.
-->

## Document control

<!-- Guidance: Tie the checklist to a specific version of the paper and code so reviewers can match answers to evidence. -->

| Field | Value |
|---|---|
| Document ID | [REPRO-ML-001] |
| Study or paper | [Title] |
| Paper version | [*Example:* arXiv v2 / submitted manuscript YYYY-MM-DD] |
| Code version | [Repository URL and commit hash or release tag] |
| Version | [0.1] |
| Status | [Draft / In review / Final] |
| Owner | [@lead-author] |
| Approvers / reviewers | [@co-author], [@internal-reviewer] |
| Scientific field | [*Example:* Computational social science] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal / Public supplement] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Lead author | [@name] | [Complete] | YYYY-MM-DD |
| Independent internal reviewer | [@name] | [Agree / Issues raised] | YYYY-MM-DD |
| Senior author | [@name] | [Approved for release] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Record each pass and the paper version it refers to. Answers often change after reviewers ask for a new baseline or split. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First pass at study design stage |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Updated after leakage audit found duplicate patients across splits |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Final, released with paper |

## Contents

<!-- Guidance: Keep aligned with the headings below. -->

1. [Module 1: Study goals](#module-1-study-goals)
2. [Module 2: Computational reproducibility](#module-2-computational-reproducibility)
3. [Module 3: Data quality](#module-3-data-quality)
4. [Module 4: Data preprocessing](#module-4-data-preprocessing)
5. [Module 5: Modelling](#module-5-modelling)
6. [Module 6: Data leakage](#module-6-data-leakage)
7. [Module 7: Metrics and uncertainty](#module-7-metrics-and-uncertainty)
8. [Module 8: Generalisability and limitations](#module-8-generalisability-and-limitations)
9. [Leakage audit worksheet](#leakage-audit-worksheet)
10. [Evaluation protocol record](#evaluation-protocol-record)
11. [Code and data release record](#code-and-data-release-record)
12. [Compute reporting](#compute-reporting)
13. [Summary and sign-off](#summary-and-sign-off)
14. [Glossary and references](#glossary-and-references)

**Answer key:** Yes (reported, with location), Partly (reported with gaps
noted), No (not reported, reason given), N/A (does not apply, reason
given).

## Module 1: Study goals

<!-- Guidance: State what the scientific claim is about and why ML is the right tool. Questions: about which population is the claim, and why that one? Pitfall: a claim about "people" supported by data from one platform or one hospital. -->

| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|---|---|---|
| 1a | Which population or data distribution the scientific claim is about | [Yes / Partly / No / N/A] | *Example:* Section 2.1: adult inpatients in two regional hospitals, 2019 to 2023 |
| 1b | Why this population or distribution was chosen | [Answer] | [Evidence] |
| 1c | Why ML methods are used to answer the question rather than other methods | [Answer] | *Example:* Section 1: nonlinear interactions among 120 lab values |

## Module 2: Computational reproducibility

<!-- Guidance: Make it possible for someone else to regenerate every reported number from released materials. Pitfall: a repository that runs only on the author's machine because paths, seeds or package versions are missing. -->

| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|---|---|---|
| 2a | The dataset used, with a persistent identifier or link (such as a DOI) to the exact version | [Answer] | *Example:* Zenodo DOI [10.xxxx/zenodo.nnnnnn], version 2 |
| 2b | The code, with a persistent identifier for the exact version used | [Answer] | *Example:* Archived release v1.0.1 with DOI |
| 2c | The computing environment and infrastructure used to produce results | [Answer] | *Example:* Section 12 of this checklist; container digest in README |
| 2d | A README that explains how to regenerate the results | [Answer] | [Evidence] |
| 2e | A script or pipeline that regenerates all reported results end to end | [Answer] | *Example:* `make reproduce` regenerates Tables 1 to 4 and Figures 2 to 5 |

## Module 3: Data quality

<!-- Guidance: Show that the data can support the claim. Questions: where does the data come from, when, how were labels made, and who is missing? Pitfall: ground-truth labels assumed correct because they came from an official system. -->

| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|---|---|---|
| 3a | Where the data came from, when it was collected, and how ground-truth labels were produced | [Answer] | [Evidence] |
| 3b | The sampling frame: the list or mechanism from which samples were drawn | [Answer] | [Evidence] |
| 3c | Why the dataset is suitable for the modelling task | [Answer] | [Evidence] |
| 3d | How the outcome variable is defined, with descriptive statistics | [Answer] | *Example:* Table S1: 30-day readmission, 11.4% positive |
| 3e | Number of samples and how often each outcome occurs | [Answer] | [Evidence] |
| 3f | How much data is missing, per variable or overall | [Answer] | [Evidence] |
| 3g | Why the dataset represents the population named in item 1a | [Answer] | [Evidence] |

## Module 4: Data preprocessing

<!-- Guidance: Record every step between raw data and model input so others can repeat it and reviewers can check it for leakage. Pitfall: silent exclusions that change the population. -->

| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|---|---|---|
| 4a | Which samples were excluded and why | [Answer] | *Example:* Figure S1 flow diagram: 48,210 to 41,977 records |
| 4b | Treatment of values that are implausible, damaged or outside valid ranges | [Answer] | [Evidence] |
| 4c | All transformations from raw data to model input (encoding, scaling, imputation, feature construction) | [Answer] | *Example:* `src/features.py`; section 3.2 |

## Module 5: Modelling

<!-- Guidance: Describe every model trained, not just the winner, and how the reported model was chosen. Pitfall: comparing a tuned new model with an untuned baseline. -->

| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|---|---|---|
| 5a | Every model trained during the study, including ones not reported in the main results | [Answer] | [Evidence] |
| 5b | Why these model types were chosen | [Answer] | [Evidence] |
| 5c | How models were evaluated: train, validation and test splits or cross-validation folds | [Answer] | *Example:* Section 10 of this checklist |
| 5d | How the reported model was selected from those trained | [Answer] | [Evidence] |
| 5e | How hyperparameters were tuned, with search spaces and budget | [Answer] | [Evidence] |
| 5f | Why the baselines are appropriate and were given a fair chance | [Answer] | *Example:* Logistic regression tuned with the same 50-trial budget |

## Module 6: Data leakage

<!-- Guidance: Leakage makes performance look better than it would be on new data. Audit for it explicitly using the worksheet in the next section. Pitfall: fitting a scaler, imputer or feature selector on the full dataset before splitting. -->

| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|---|---|---|
| 6a | Preprocessing, feature selection and model fitting use only training data, never test data | [Answer] | *Example:* Pipelines fitted inside each fold; `src/pipeline.py` |
| 6b | How dependencies or duplicates between training and test data were handled (same person, same site, near-duplicates, time order) | [Answer] | *Example:* Grouped split by patient ID; temporal split by admission date |
| 6c | Why each feature is legitimate: available at prediction time and not a proxy for the outcome | [Answer] | *Example:* Leakage audit worksheet, rows LEAK-003 and LEAK-004 |

## Module 7: Metrics and uncertainty

<!-- Guidance: Choose metrics that match the claim and report how uncertain each estimate is. Pitfall: accuracy on an imbalanced outcome, or a single-run number with no interval. -->

| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|---|---|---|
| 7a | Each performance metric and why it fits the task and claim | [Answer] | [Evidence] |
| 7b | Uncertainty estimates (for example confidence intervals) and how they were computed | [Answer] | *Example:* 95% bootstrap CI over test patients, 1,000 resamples |
| 7c | Why any statistical tests used are appropriate and whether their assumptions hold | [Answer] | [Evidence] |

## Module 8: Generalisability and limitations

<!-- Guidance: Say where the result should and should not be expected to hold. Pitfall: generalising from an internal test split to other sites or periods. -->

| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|---|---|---|
| 8a | Evidence that the findings hold outside the study data (external validity) | [Answer] | *Example:* External test on a third hospital, AUROC 0.79 versus 0.83 internal |
| 8b | Settings where the authors would not expect the results to apply | [Answer] | [Evidence] |

## Leakage audit worksheet

<!-- Guidance: Walk through each leakage pattern below for your pipeline. Record what you checked and how. These patterns are common categories discussed in the leakage literature (see references), described here in original words. Pitfall: checking only the obvious train-test overlap. -->

| ID | Leakage pattern | Question to ask | Check performed | Result | Fix |
|---|---|---|---|---|---|
| LEAK-001 | No separate test set | Was any reported number computed on data used for training or tuning? | *Example:* Traced every table to its data split | None found | Not applicable |
| LEAK-002 | Preprocessing on all data | Were scaling, imputation, resampling or feature selection fitted before the split? | *Example:* Code review of `pipeline.py` | Found: SMOTE applied before split | Moved inside folds; Table 2 updated |
| LEAK-003 | Feature not available at prediction time | Would each feature exist at the moment a real prediction is made? | *Example:* Feature timeline review with clinician | Found: discharge medication list | Removed |
| LEAK-004 | Proxy for the outcome | Does any feature encode the label indirectly? | *Example:* Single-feature models; inspect top importances | None found | Not applicable |
| LEAK-005 | Duplicates or related samples across splits | Can the same entity, or near-identical items, appear in train and test? | *Example:* Grouped split by patient; hash near-duplicate notes | None after fix | Grouped split |
| LEAK-006 | Temporal leakage | Does training data come from after the test data in time? | *Example:* Split by admission date | None found | Not applicable |
| LEAK-007 | Sampling bias in test distribution | Is the test set drawn from the population the claim targets? | [Check] | [Result] | [Fix] |
| LEAK-008 | Test set reused for model selection | Was the test set consulted while choosing models or thresholds? | [Check] | [Result] | [Fix] |

```mermaid
flowchart LR
  R[Raw data] --> S{Split first<br/>grouped and temporal}
  S --> TR[Training data]
  S --> TE[Test data<br/>locked]
  TR --> P[Fit preprocessing<br/>and features]
  P --> M[Train and tune<br/>inner validation]
  M --> F[Final model]
  TE --> E[Evaluate once]
  F --> E
```

## Evaluation protocol record

<!-- Guidance: Fix the protocol before looking at test results and record any deviation. This section supports items 5c, 5d, 7a and 7b. Pitfall: changing the metric or threshold after seeing which one makes the result look best. -->

| Element | Value |
|---|---|
| Split strategy | [*Example:* Grouped by patient, temporal: train 2019 to 2021, validation 2022, test 2023] |
| Cross-validation | [*Example:* 5-fold grouped CV on training data for tuning only] |
| Test set locked on | YYYY-MM-DD, by [@name] |
| Primary metric | [*Example:* AUROC] |
| Secondary metrics | [*Example:* Calibration slope, sensitivity at 90% specificity] |
| Uncertainty method | [*Example:* Patient-level bootstrap, 1,000 resamples] |
| Baselines | [*Example:* Logistic regression, existing clinical score] |
| Seeds | [*Example:* 5 seeds; mean and standard deviation reported] |
| Pre-registration | [Link or "not pre-registered"] |
| Deviations from protocol | [List with reason, or "None"] |

## Code and data release record

<!-- Guidance: Record what is released, where, under which licence and with what restrictions. This section supports items 2a, 2b, 2d and 2e. Pitfall: personal data released inside a "convenience" copy of the processed dataset. -->

| Artifact | Location | Persistent ID | Licence | Access conditions | Status |
|---|---|---|---|---|---|
| Code *(Example)* | [Repository URL] | [Archive DOI] | MIT | Open | Released |
| Processed features *(Example)* | [Repository] | [DOI] | CC BY 4.0 | Open, de-identified | Released |
| Raw data *(Example)* | Hospital data office | Not applicable | Data use agreement | Application required | Restricted |
| Trained model *(Example)* | [Model hub URL] | [DOI] | [Licence] | Open | Released |
| Evaluation outputs and predictions *(Example)* | [Repository] | [DOI] | CC BY 4.0 | Open | Released |
| [Artifact] | [Location] | [ID] | [Licence] | [Conditions] | [Status] |

If data cannot be shared, state why and describe how others can obtain
access, or provide a synthetic or sample dataset that exercises the full
pipeline.

## Compute reporting

<!-- Guidance: Report hardware, time and total compute, including tuning and failed runs. This supports item 2c and helps readers judge the cost of reproduction. Pitfall: reporting only the final training run. -->

| Phase | Hardware | Software environment | Wall-clock time | Runs | Total compute |
|---|---|---|---|---|---|
| Hyperparameter search *(Example)* | 1 x [GPU model], 24 GB | Python 3.12, [framework and version] | 0.5 h per trial | 50 trials | 25 GPU-hours |
| Final training *(Example)* | Same | Same | 2 h | 5 seeds | 10 GPU-hours |
| Bootstrap evaluation *(Example)* | 32-core CPU | Same | 1 h | 1 | 32 CPU-hours |
| Exploratory and failed runs *(Example)* | Same | Same | Various | Various | about 60 GPU-hours (estimate) |
| [Phase] | [Hardware] | [Environment] | [Time] | [Runs] | [Total] |

**Estimated cost to reproduce the main results:** [*Example:* About 12
GPU-hours on one 24 GB GPU, excluding the hyperparameter search.]

## Summary and sign-off

<!-- Guidance: Count answers per module and list open issues that a reader should know about. Pitfall: signing off with items answered No and no explanation. -->

| Module | Items | Yes | Partly | No | N/A |
|---|---|---|---|---|---|
| 1 Study goals | 3 | [n] | [n] | [n] | [n] |
| 2 Computational reproducibility | 5 | [n] | [n] | [n] | [n] |
| 3 Data quality | 7 | [n] | [n] | [n] | [n] |
| 4 Data preprocessing | 3 | [n] | [n] | [n] | [n] |
| 5 Modelling | 6 | [n] | [n] | [n] | [n] |
| 6 Data leakage | 3 | [n] | [n] | [n] | [n] |
| 7 Metrics and uncertainty | 3 | [n] | [n] | [n] | [n] |
| 8 Generalisability and limitations | 2 | [n] | [n] | [n] | [n] |
| **Total** | **32** | [n] | [n] | [n] | [n] |

**Open issues:**

| ID | Item | Issue | Planned action | Owner |
|---|---|---|---|---|
| REPRO-ISS-001 *(Example)* | 8a | No external validation yet | Request data from third site | [@lead-author] |
| [ID] | [Item] | [Issue] | [Action] | [@owner] |

**Independent reproduction:** [*Example:* A co-author not involved in the
analysis ran `make reproduce` from a clean clone on YYYY-MM-DD and matched
all numbers in Tables 1 to 4 within reported intervals.]

## Glossary and references

<!-- Guidance: Define terms that differ between fields and list the sources. -->

### Glossary

| Term | Meaning in this checklist |
|---|---|
| Computational reproducibility | Getting the same results from the same data and code |
| Data leakage | Information from outside the training data that inflates measured performance |
| External validity | Whether findings hold for populations or settings beyond the study data |
| Grouped split | A split that keeps all samples from one entity (person, site, document) on one side |
| ML-based science | Research that uses machine learning models to support a scientific claim |
| Sampling frame | The list or mechanism from which study samples are drawn |

### References

- Kapoor, S., Cantrell, E. M., Peng, K., et al. (2024). REFORMS:
  Consensus-based Recommendations for Machine-learning-based Science.
  Science Advances, 10(18), eadk3452.
  https://doi.org/10.1126/sciadv.adk3452
- REFORMS project page with checklist and guidelines documents:
  https://reforms.cs.princeton.edu/
- Kapoor, S. and Narayanan, A. (2023). Leakage and the reproducibility
  crisis in machine-learning-based science. Patterns, 4(9), 100804.
  https://doi.org/10.1016/j.patter.2023.100804
- Related templates: Reproducibility Checklist
  (`science/reproducibility-checklist`), ML Conference Paper
  (`academic/ml-conference-paper`), Datasheet for Datasets
  (`ai-data/datasheet-for-datasets`), Model Card (`ai-data/model-card`).

## Standard and licence note

Structure aligned with REFORMS: Consensus-based Recommendations for
Machine-learning-based Science (Kapoor et al., Science Advances, 2024),
including its 8 modules and 32 item numbers (1a to 8b). The article is
published under CC BY-NC 4.0; this template does not reproduce its
checklist wording or guidelines. Every item description, worksheet and
example here is original text, and example numbers are illustrative.
Check the REFORMS project page for the current checklist and guidelines
before use. Template licence: MIT.