ML Reproducibility Checklist Template (REFORMS)

REFORMS-aligned ML science checklist: 32 items in 8 modules, leakage audit worksheet, evaluation protocol, code and data release, compute reporting

AI & DataFullREFORMS (Kapoor et al., Science Advances 2024), 32 items in 8 modules, paraphrasedreproducibility checklistUpdated Oct 22, 2026

Open in editorDownload .md

How to use: Use it when machine learning supports a scientific claim, at study design and again before submission or release. Fill the leakage audit worksheet and evaluation protocol first, then answer the 32 items with evidence pointers. The main pitfall is treating reproducible numbers as valid ones when the pipeline leaks.

Preview

ML Reproducibility and Validity Checklist: [Study or paper title]

Reproducibility and validity checklist for research that uses machine learning to make a scientific claim. Structure aligned with REFORMS (Kapoor et al., Science Advances, 2024): 32 items in 8 modules, here paraphrased in original words, with answer and evidence fields, plus working sections on data leakage, evaluation, code and data release, and compute reporting. Replace every [bracketed] value and every row marked Example.

Document control

Field	Value
Document ID	[REPRO-ML-001]
Study or paper	[Title]
Paper version	[Example: arXiv v2 / submitted manuscript YYYY-MM-DD]
Code version	[Repository URL and commit hash or release tag]
Version	[0.1]
Status	[Draft / In review / Final]
Owner	[@lead-author]
Approvers / reviewers	[@co-author], [@internal-reviewer]
Scientific field	[Example: Computational social science]
Last updated	YYYY-MM-DD
Classification	[Internal / Public supplement]

Approvals

Role	Name	Decision	Date
Lead author	[@name]	[Complete]	YYYY-MM-DD
Independent internal reviewer	[@name]	[Agree / Issues raised]	YYYY-MM-DD
Senior author	[@name]	[Approved for release]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First pass at study design stage
0.2	YYYY-MM-DD	[@name]	Example: Updated after leakage audit found duplicate patients across splits
1.0	YYYY-MM-DD	[@name]	Example: Final, released with paper

Module 1: Study goals
Module 2: Computational reproducibility
Module 3: Data quality
Module 4: Data preprocessing
Module 5: Modelling
Module 6: Data leakage
Module 7: Metrics and uncertainty
Module 8: Generalisability and limitations
Leakage audit worksheet
Evaluation protocol record
Code and data release record
Compute reporting
Summary and sign-off
Glossary and references

Answer key: Yes (reported, with location), Partly (reported with gaps noted), No (not reported, reason given), N/A (does not apply, reason given).

Module 1: Study goals

Item	What to report (paraphrased)	Answer	Evidence (section, file)
1a	Which population or data distribution the scientific claim is about	[Yes / Partly / No / N/A]	Example: Section 2.1: adult inpatients in two regional hospitals, 2019 to 2023
1b	Why this population or distribution was chosen	[Answer]	[Evidence]
1c	Why ML methods are used to answer the question rather than other methods	[Answer]	Example: Section 1: nonlinear interactions among 120 lab values

Module 2: Computational reproducibility

Item	What to report (paraphrased)	Answer	Evidence (section, file)
2a	The dataset used, with a persistent identifier or link (such as a DOI) to the exact version	[Answer]	Example: Zenodo DOI [10.xxxx/zenodo.nnnnnn], version 2
2b	The code, with a persistent identifier for the exact version used	[Answer]	Example: Archived release v1.0.1 with DOI
2c	The computing environment and infrastructure used to produce results	[Answer]	Example: Section 12 of this checklist; container digest in README
2d	A README that explains how to regenerate the results	[Answer]	[Evidence]
2e	A script or pipeline that regenerates all reported results end to end	[Answer]	Example: `make reproduce` regenerates Tables 1 to 4 and Figures 2 to 5

Module 3: Data quality

Item	What to report (paraphrased)	Answer	Evidence (section, file)
3a	Where the data came from, when it was collected, and how ground-truth labels were produced	[Answer]	[Evidence]
3b	The sampling frame: the list or mechanism from which samples were drawn	[Answer]	[Evidence]
3c	Why the dataset is suitable for the modelling task	[Answer]	[Evidence]
3d	How the outcome variable is defined, with descriptive statistics	[Answer]	Example: Table S1: 30-day readmission, 11.4% positive
3e	Number of samples and how often each outcome occurs	[Answer]	[Evidence]
3f	How much data is missing, per variable or overall	[Answer]	[Evidence]
3g	Why the dataset represents the population named in item 1a	[Answer]	[Evidence]

Module 4: Data preprocessing

Item	What to report (paraphrased)	Answer	Evidence (section, file)
4a	Which samples were excluded and why	[Answer]	Example: Figure S1 flow diagram: 48,210 to 41,977 records
4b	Treatment of values that are implausible, damaged or outside valid ranges	[Answer]	[Evidence]
4c	All transformations from raw data to model input (encoding, scaling, imputation, feature construction)	[Answer]	Example: `src/features.py`; section 3.2

Module 5: Modelling

Item	What to report (paraphrased)	Answer	Evidence (section, file)
5a	Every model trained during the study, including ones not reported in the main results	[Answer]	[Evidence]
5b	Why these model types were chosen	[Answer]	[Evidence]
5c	How models were evaluated: train, validation and test splits or cross-validation folds	[Answer]	Example: Section 10 of this checklist
5d	How the reported model was selected from those trained	[Answer]	[Evidence]
5e	How hyperparameters were tuned, with search spaces and budget	[Answer]	[Evidence]
5f	Why the baselines are appropriate and were given a fair chance	[Answer]	Example: Logistic regression tuned with the same 50-trial budget

Module 6: Data leakage

Item	What to report (paraphrased)	Answer	Evidence (section, file)
6a	Preprocessing, feature selection and model fitting use only training data, never test data	[Answer]	Example: Pipelines fitted inside each fold; `src/pipeline.py`
6b	How dependencies or duplicates between training and test data were handled (same person, same site, near-duplicates, time order)	[Answer]	Example: Grouped split by patient ID; temporal split by admission date
6c	Why each feature is legitimate: available at prediction time and not a proxy for the outcome	[Answer]	Example: Leakage audit worksheet, rows LEAK-003 and LEAK-004

Module 7: Metrics and uncertainty

Item	What to report (paraphrased)	Answer	Evidence (section, file)
7a	Each performance metric and why it fits the task and claim	[Answer]	[Evidence]
7b	Uncertainty estimates (for example confidence intervals) and how they were computed	[Answer]	Example: 95% bootstrap CI over test patients, 1,000 resamples
7c	Why any statistical tests used are appropriate and whether their assumptions hold	[Answer]	[Evidence]

Module 8: Generalisability and limitations

Item	What to report (paraphrased)	Answer	Evidence (section, file)
8a	Evidence that the findings hold outside the study data (external validity)	[Answer]	Example: External test on a third hospital, AUROC 0.79 versus 0.83 internal
8b	Settings where the authors would not expect the results to apply	[Answer]	[Evidence]

Leakage audit worksheet

ID	Leakage pattern	Question to ask	Check performed	Result	Fix
LEAK-001	No separate test set	Was any reported number computed on data used for training or tuning?	Example: Traced every table to its data split	None found	Not applicable
LEAK-002	Preprocessing on all data	Were scaling, imputation, resampling or feature selection fitted before the split?	Example: Code review of `pipeline.py`	Found: SMOTE applied before split	Moved inside folds; Table 2 updated
LEAK-003	Feature not available at prediction time	Would each feature exist at the moment a real prediction is made?	Example: Feature timeline review with clinician	Found: discharge medication list	Removed
LEAK-004	Proxy for the outcome	Does any feature encode the label indirectly?	Example: Single-feature models; inspect top importances	None found	Not applicable
LEAK-005	Duplicates or related samples across splits	Can the same entity, or near-identical items, appear in train and test?	Example: Grouped split by patient; hash near-duplicate notes	None after fix	Grouped split
LEAK-006	Temporal leakage	Does training data come from after the test data in time?	Example: Split by admission date	None found	Not applicable
LEAK-007	Sampling bias in test distribution	Is the test set drawn from the population the claim targets?	[Check]	[Result]	[Fix]
LEAK-008	Test set reused for model selection	Was the test set consulted while choosing models or thresholds?	[Check]	[Result]	[Fix]

Evaluation protocol record

Element	Value
Split strategy	[Example: Grouped by patient, temporal: train 2019 to 2021, validation 2022, test 2023]
Cross-validation	[Example: 5-fold grouped CV on training data for tuning only]
Test set locked on	YYYY-MM-DD, by [@name]
Primary metric	[Example: AUROC]
Secondary metrics	[Example: Calibration slope, sensitivity at 90% specificity]
Uncertainty method	[Example: Patient-level bootstrap, 1,000 resamples]
Baselines	[Example: Logistic regression, existing clinical score]
Seeds	[Example: 5 seeds; mean and standard deviation reported]
Pre-registration	[Link or "not pre-registered"]
Deviations from protocol	[List with reason, or "None"]

Code and data release record

Artifact	Location	Persistent ID	Licence	Access conditions	Status
Code (Example)	[Repository URL]	[Archive DOI]	MIT	Open	Released
Processed features (Example)	[Repository]	[DOI]	CC BY 4.0	Open, de-identified	Released
Raw data (Example)	Hospital data office	Not applicable	Data use agreement	Application required	Restricted
Trained model (Example)	[Model hub URL]	[DOI]	[Licence]	Open	Released
Evaluation outputs and predictions (Example)	[Repository]	[DOI]	CC BY 4.0	Open	Released
[Artifact]	[Location]	[ID]	[Licence]	[Conditions]	[Status]

If data cannot be shared, state why and describe how others can obtain access, or provide a synthetic or sample dataset that exercises the full pipeline.

Compute reporting

Phase	Hardware	Software environment	Wall-clock time	Runs	Total compute
Hyperparameter search (Example)	1 x [GPU model], 24 GB	Python 3.12, [framework and version]	0.5 h per trial	50 trials	25 GPU-hours
Final training (Example)	Same	Same	2 h	5 seeds	10 GPU-hours
Bootstrap evaluation (Example)	32-core CPU	Same	1 h	1	32 CPU-hours
Exploratory and failed runs (Example)	Same	Same	Various	Various	about 60 GPU-hours (estimate)
[Phase]	[Hardware]	[Environment]	[Time]	[Runs]	[Total]

Estimated cost to reproduce the main results: [Example: About 12 GPU-hours on one 24 GB GPU, excluding the hyperparameter search.]

Summary and sign-off

Module	Items	Yes	Partly	No	N/A
1 Study goals	3	[n]	[n]	[n]	[n]
2 Computational reproducibility	5	[n]	[n]	[n]	[n]
3 Data quality	7	[n]	[n]	[n]	[n]
4 Data preprocessing	3	[n]	[n]	[n]	[n]
5 Modelling	6	[n]	[n]	[n]	[n]
6 Data leakage	3	[n]	[n]	[n]	[n]
7 Metrics and uncertainty	3	[n]	[n]	[n]	[n]
8 Generalisability and limitations	2	[n]	[n]	[n]	[n]
Total	32	[n]	[n]	[n]	[n]

Open issues:

ID	Item	Issue	Planned action	Owner
REPRO-ISS-001 (Example)	8a	No external validation yet	Request data from third site	[@lead-author]
[ID]	[Item]	[Issue]	[Action]	[@owner]

Independent reproduction: [Example: A co-author not involved in the analysis ran make reproduce from a clean clone on YYYY-MM-DD and matched all numbers in Tables 1 to 4 within reported intervals.]

Glossary and references

Glossary

Term	Meaning in this checklist
Computational reproducibility	Getting the same results from the same data and code
Data leakage	Information from outside the training data that inflates measured performance
External validity	Whether findings hold for populations or settings beyond the study data
Grouped split	A split that keeps all samples from one entity (person, site, document) on one side
ML-based science	Research that uses machine learning models to support a scientific claim
Sampling frame	The list or mechanism from which study samples are drawn

References

Kapoor, S., Cantrell, E. M., Peng, K., et al. (2024). REFORMS: Consensus-based Recommendations for Machine-learning-based Science. Science Advances, 10(18), eadk3452. https://doi.org/10.1126/sciadv.adk3452
REFORMS project page with checklist and guidelines documents: https://reforms.cs.princeton.edu/
Kapoor, S. and Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804
Related templates: Reproducibility Checklist (science/reproducibility-checklist), ML Conference Paper (academic/ml-conference-paper), Datasheet for Datasets (ai-data/datasheet-for-datasets), Model Card (ai-data/model-card).

Standard and licence note

Structure aligned with REFORMS: Consensus-based Recommendations for Machine-learning-based Science (Kapoor et al., Science Advances, 2024), including its 8 modules and 32 item numbers (1a to 8b). The article is published under CC BY-NC 4.0; this template does not reproduce its checklist wording or guidelines. Every item description, worksheet and example here is original text, and example numbers are illustrative. Check the REFORMS project page for the current checklist and guidelines before use. Template licence: MIT.

Related templates

AI & Data