ML Reproducibility and Validity Checklist: [Study or paper title]
Reproducibility and validity checklist for research that uses machine
learning to make a scientific claim. Structure aligned with REFORMS
(Kapoor et al., Science Advances, 2024): 32 items in 8 modules, here
paraphrased in original words, with answer and evidence fields, plus
working sections on data leakage, evaluation, code and data release,
and compute reporting. Replace every [bracketed] value and every row
marked Example.
Document control
| Field | Value |
|---|
| Document ID | [REPRO-ML-001] |
| Study or paper | [Title] |
| Paper version | [Example: arXiv v2 / submitted manuscript YYYY-MM-DD] |
| Code version | [Repository URL and commit hash or release tag] |
| Version | [0.1] |
| Status | [Draft / In review / Final] |
| Owner | [@lead-author] |
| Approvers / reviewers | [@co-author], [@internal-reviewer] |
| Scientific field | [Example: Computational social science] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal / Public supplement] |
Approvals
| Role | Name | Decision | Date |
|---|
| Lead author | [@name] | [Complete] | YYYY-MM-DD |
| Independent internal reviewer | [@name] | [Agree / Issues raised] | YYYY-MM-DD |
| Senior author | [@name] | [Approved for release] | YYYY-MM-DD |
Revision history
| Version | Date | Author | Change summary |
|---|
| 0.1 | YYYY-MM-DD | [@name] | First pass at study design stage |
| 0.2 | YYYY-MM-DD | [@name] | Example: Updated after leakage audit found duplicate patients across splits |
| 1.0 | YYYY-MM-DD | [@name] | Example: Final, released with paper |
Contents
Answer key: Yes (reported, with location), Partly (reported with gaps
noted), No (not reported, reason given), N/A (does not apply, reason
given).
Module 1: Study goals
| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|
| 1a | Which population or data distribution the scientific claim is about | [Yes / Partly / No / N/A] | Example: Section 2.1: adult inpatients in two regional hospitals, 2019 to 2023 |
| 1b | Why this population or distribution was chosen | [Answer] | [Evidence] |
| 1c | Why ML methods are used to answer the question rather than other methods | [Answer] | Example: Section 1: nonlinear interactions among 120 lab values |
Module 2: Computational reproducibility
| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|
| 2a | The dataset used, with a persistent identifier or link (such as a DOI) to the exact version | [Answer] | Example: Zenodo DOI [10.xxxx/zenodo.nnnnnn], version 2 |
| 2b | The code, with a persistent identifier for the exact version used | [Answer] | Example: Archived release v1.0.1 with DOI |
| 2c | The computing environment and infrastructure used to produce results | [Answer] | Example: Section 12 of this checklist; container digest in README |
| 2d | A README that explains how to regenerate the results | [Answer] | [Evidence] |
| 2e | A script or pipeline that regenerates all reported results end to end | [Answer] | Example: make reproduce regenerates Tables 1 to 4 and Figures 2 to 5 |
Module 3: Data quality
| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|
| 3a | Where the data came from, when it was collected, and how ground-truth labels were produced | [Answer] | [Evidence] |
| 3b | The sampling frame: the list or mechanism from which samples were drawn | [Answer] | [Evidence] |
| 3c | Why the dataset is suitable for the modelling task | [Answer] | [Evidence] |
| 3d | How the outcome variable is defined, with descriptive statistics | [Answer] | Example: Table S1: 30-day readmission, 11.4% positive |
| 3e | Number of samples and how often each outcome occurs | [Answer] | [Evidence] |
| 3f | How much data is missing, per variable or overall | [Answer] | [Evidence] |
| 3g | Why the dataset represents the population named in item 1a | [Answer] | [Evidence] |
Module 4: Data preprocessing
| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|
| 4a | Which samples were excluded and why | [Answer] | Example: Figure S1 flow diagram: 48,210 to 41,977 records |
| 4b | Treatment of values that are implausible, damaged or outside valid ranges | [Answer] | [Evidence] |
| 4c | All transformations from raw data to model input (encoding, scaling, imputation, feature construction) | [Answer] | Example: src/features.py; section 3.2 |
Module 5: Modelling
| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|
| 5a | Every model trained during the study, including ones not reported in the main results | [Answer] | [Evidence] |
| 5b | Why these model types were chosen | [Answer] | [Evidence] |
| 5c | How models were evaluated: train, validation and test splits or cross-validation folds | [Answer] | Example: Section 10 of this checklist |
| 5d | How the reported model was selected from those trained | [Answer] | [Evidence] |
| 5e | How hyperparameters were tuned, with search spaces and budget | [Answer] | [Evidence] |
| 5f | Why the baselines are appropriate and were given a fair chance | [Answer] | Example: Logistic regression tuned with the same 50-trial budget |
Module 6: Data leakage
| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|
| 6a | Preprocessing, feature selection and model fitting use only training data, never test data | [Answer] | Example: Pipelines fitted inside each fold; src/pipeline.py |
| 6b | How dependencies or duplicates between training and test data were handled (same person, same site, near-duplicates, time order) | [Answer] | Example: Grouped split by patient ID; temporal split by admission date |
| 6c | Why each feature is legitimate: available at prediction time and not a proxy for the outcome | [Answer] | Example: Leakage audit worksheet, rows LEAK-003 and LEAK-004 |
Module 7: Metrics and uncertainty
| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|
| 7a | Each performance metric and why it fits the task and claim | [Answer] | [Evidence] |
| 7b | Uncertainty estimates (for example confidence intervals) and how they were computed | [Answer] | Example: 95% bootstrap CI over test patients, 1,000 resamples |
| 7c | Why any statistical tests used are appropriate and whether their assumptions hold | [Answer] | [Evidence] |
Module 8: Generalisability and limitations
| Item | What to report (paraphrased) | Answer | Evidence (section, file) |
|---|
| 8a | Evidence that the findings hold outside the study data (external validity) | [Answer] | Example: External test on a third hospital, AUROC 0.79 versus 0.83 internal |
| 8b | Settings where the authors would not expect the results to apply | [Answer] | [Evidence] |
Leakage audit worksheet
| ID | Leakage pattern | Question to ask | Check performed | Result | Fix |
|---|
| LEAK-001 | No separate test set | Was any reported number computed on data used for training or tuning? | Example: Traced every table to its data split | None found | Not applicable |
| LEAK-002 | Preprocessing on all data | Were scaling, imputation, resampling or feature selection fitted before the split? | Example: Code review of pipeline.py | Found: SMOTE applied before split | Moved inside folds; Table 2 updated |
| LEAK-003 | Feature not available at prediction time | Would each feature exist at the moment a real prediction is made? | Example: Feature timeline review with clinician | Found: discharge medication list | Removed |
| LEAK-004 | Proxy for the outcome | Does any feature encode the label indirectly? | Example: Single-feature models; inspect top importances | None found | Not applicable |
| LEAK-005 | Duplicates or related samples across splits | Can the same entity, or near-identical items, appear in train and test? | Example: Grouped split by patient; hash near-duplicate notes | None after fix | Grouped split |
| LEAK-006 | Temporal leakage | Does training data come from after the test data in time? | Example: Split by admission date | None found | Not applicable |
| LEAK-007 | Sampling bias in test distribution | Is the test set drawn from the population the claim targets? | [Check] | [Result] | [Fix] |
| LEAK-008 | Test set reused for model selection | Was the test set consulted while choosing models or thresholds? | [Check] | [Result] | [Fix] |
Evaluation protocol record
| Element | Value |
|---|
| Split strategy | [Example: Grouped by patient, temporal: train 2019 to 2021, validation 2022, test 2023] |
| Cross-validation | [Example: 5-fold grouped CV on training data for tuning only] |
| Test set locked on | YYYY-MM-DD, by [@name] |
| Primary metric | [Example: AUROC] |
| Secondary metrics | [Example: Calibration slope, sensitivity at 90% specificity] |
| Uncertainty method | [Example: Patient-level bootstrap, 1,000 resamples] |
| Baselines | [Example: Logistic regression, existing clinical score] |
| Seeds | [Example: 5 seeds; mean and standard deviation reported] |
| Pre-registration | [Link or "not pre-registered"] |
| Deviations from protocol | [List with reason, or "None"] |
Code and data release record
| Artifact | Location | Persistent ID | Licence | Access conditions | Status |
|---|
| Code (Example) | [Repository URL] | [Archive DOI] | MIT | Open | Released |
| Processed features (Example) | [Repository] | [DOI] | CC BY 4.0 | Open, de-identified | Released |
| Raw data (Example) | Hospital data office | Not applicable | Data use agreement | Application required | Restricted |
| Trained model (Example) | [Model hub URL] | [DOI] | [Licence] | Open | Released |
| Evaluation outputs and predictions (Example) | [Repository] | [DOI] | CC BY 4.0 | Open | Released |
| [Artifact] | [Location] | [ID] | [Licence] | [Conditions] | [Status] |
If data cannot be shared, state why and describe how others can obtain
access, or provide a synthetic or sample dataset that exercises the full
pipeline.
Compute reporting
| Phase | Hardware | Software environment | Wall-clock time | Runs | Total compute |
|---|
| Hyperparameter search (Example) | 1 x [GPU model], 24 GB | Python 3.12, [framework and version] | 0.5 h per trial | 50 trials | 25 GPU-hours |
| Final training (Example) | Same | Same | 2 h | 5 seeds | 10 GPU-hours |
| Bootstrap evaluation (Example) | 32-core CPU | Same | 1 h | 1 | 32 CPU-hours |
| Exploratory and failed runs (Example) | Same | Same | Various | Various | about 60 GPU-hours (estimate) |
| [Phase] | [Hardware] | [Environment] | [Time] | [Runs] | [Total] |
Estimated cost to reproduce the main results: [Example: About 12
GPU-hours on one 24 GB GPU, excluding the hyperparameter search.]
Summary and sign-off
| Module | Items | Yes | Partly | No | N/A |
|---|
| 1 Study goals | 3 | [n] | [n] | [n] | [n] |
| 2 Computational reproducibility | 5 | [n] | [n] | [n] | [n] |
| 3 Data quality | 7 | [n] | [n] | [n] | [n] |
| 4 Data preprocessing | 3 | [n] | [n] | [n] | [n] |
| 5 Modelling | 6 | [n] | [n] | [n] | [n] |
| 6 Data leakage | 3 | [n] | [n] | [n] | [n] |
| 7 Metrics and uncertainty | 3 | [n] | [n] | [n] | [n] |
| 8 Generalisability and limitations | 2 | [n] | [n] | [n] | [n] |
| Total | 32 | [n] | [n] | [n] | [n] |
| ID | Item | Issue | Planned action | Owner |
|---|
| REPRO-ISS-001 (Example) | 8a | No external validation yet | Request data from third site | [@lead-author] |
| [ID] | [Item] | [Issue] | [Action] | [@owner] |
Independent reproduction: [Example: A co-author not involved in the
analysis ran make reproduce from a clean clone on YYYY-MM-DD and matched
all numbers in Tables 1 to 4 within reported intervals.]
Glossary and references
Glossary
| Term | Meaning in this checklist |
|---|
| Computational reproducibility | Getting the same results from the same data and code |
| Data leakage | Information from outside the training data that inflates measured performance |
| External validity | Whether findings hold for populations or settings beyond the study data |
| Grouped split | A split that keeps all samples from one entity (person, site, document) on one side |
| ML-based science | Research that uses machine learning models to support a scientific claim |
| Sampling frame | The list or mechanism from which study samples are drawn |
References
- Kapoor, S., Cantrell, E. M., Peng, K., et al. (2024). REFORMS:
Consensus-based Recommendations for Machine-learning-based Science.
Science Advances, 10(18), eadk3452.
https://doi.org/10.1126/sciadv.adk3452
- REFORMS project page with checklist and guidelines documents:
https://reforms.cs.princeton.edu/
- Kapoor, S. and Narayanan, A. (2023). Leakage and the reproducibility
crisis in machine-learning-based science. Patterns, 4(9), 100804.
https://doi.org/10.1016/j.patter.2023.100804
- Related templates: Reproducibility Checklist
(
science/reproducibility-checklist), ML Conference Paper
(academic/ml-conference-paper), Datasheet for Datasets
(ai-data/datasheet-for-datasets), Model Card (ai-data/model-card).
Standard and licence note
Structure aligned with REFORMS: Consensus-based Recommendations for
Machine-learning-based Science (Kapoor et al., Science Advances, 2024),
including its 8 modules and 32 item numbers (1a to 8b). The article is
published under CC BY-NC 4.0; this template does not reproduce its
checklist wording or guidelines. Every item description, worksheet and
example here is original text, and example numbers are illustrative.
Check the REFORMS project page for the current checklist and guidelines
before use. Template licence: MIT.