Computational Reproducibility Checklist Template

Reproducibility checklist for computational research: code, environment, data, seeds, hyperparameters, compute, variance

Наукові дослідженняСтандартreproducibility checklist mathОновлено 20 жовт. 2026 р.

Відкрити в редакторіЗавантажити .md

Як використовувати: Use before submitting a computational or ML paper or releasing an artefact. Fill the summary table with an evidence link for every Yes. Main pitfall: claiming reproducibility without pinned versions or an independent rerun.

Перегляд

Reproducibility Checklist: [Paper or project title]

Checklist for computational research, including machine learning and data analysis studies, that records the evidence another team needs to rerun the work and obtain the same results: code, environment, data, randomness, hyperparameters, compute, statistical reporting, evaluation and limitations. Based on common practice in research software and conference reproducibility checklists. Replace every [bracketed] value and every row marked Example.

Field	Value
Checklist ID	[REPRO-2026-007]
Paper / project	[Title, venue or DOI]
Repository	[URL and commit hash or release tag]
Version	[1.0]
Status	[Draft / Complete / Independently verified]
Completed by	[@name]
Verified by	[@name, or "not yet verified"]
Last updated	YYYY-MM-DD

1. Summary checklist

The table summarises the status of each area; details follow in later sections.

ID	Item	Status	Evidence link	Section
RC-01	Source code publicly available	[Yes]	Example: [github.com/org/repo at tag v1.2.0]	2
RC-02	Licence for code stated	[Yes]	[LICENSE file]	2
RC-03	Software environment fully specified	[Partial]	[environment.yml; CUDA version missing]	3
RC-04	Data available or access route described	[Yes]	[Zenodo DOI]	4
RC-05	Random seeds fixed and reported	[Yes]	[config/seeds.yaml]	5
RC-06	All hyperparameters and search ranges reported	[Yes]	[Appendix B]	6
RC-07	Compute budget reported	[Yes]	[Section 7 table]	7
RC-08	Variance across runs reported	[Yes]	[Table 2, 5 seeds]	8
RC-09	Evaluation protocol fully described	[Yes]	[eval/README.md]	9
RC-10	Limitations stated	[Yes]	[Paper section 6]	10
RC-11	One-command reproduction of main result	[Partial]	[make reproduce-table2]	2

2. Code availability

Code is available at [repository URL], release [tag], archived at [DOI].

Result in paper	Script or command	Expected runtime
Example: Table 2	`python scripts/train.py --config configs/table2.yaml && python scripts/eval.py`	6 GPU hours
Figure 3	[Command]	[Runtime]
[Result]	[Command]	[Runtime]

Code licence stated: [MIT / Apache-2.0 / other]
Pre-trained models or checkpoints released: [link, or reason not]
Instructions tested on a clean machine: [date, by whom]

3. Environment specification

Component	Version	How pinned
Operating system	[Ubuntu 22.04]	[Container base image]
Language	[Python 3.11.9]	[.python-version]
Key libraries	Example: torch 2.4.1, numpy 1.26.4, scikit-learn 1.5.1	requirements.lock
GPU driver / CUDA / cuDNN	[Version]	[Documented]
Container	[Image name]	[Digest sha256:...]

4. Data availability

Dataset	Version / date	Source and licence	Preprocessing	Splits
Example: [Dataset name]	v2.1, downloaded YYYY-MM-DD	[URL], CC BY 4.0	`scripts/prep.py`, removes duplicates	Official train/val/test
[Dataset]	[Version]	[Source, licence]	[Steps]	[Split method and seed]

Restricted data: [what cannot be shared, why, and the access process]. Personal data handling: [anonymisation, consent, applicable law].

5. Randomness and seeds

Source of randomness	Seed or control	Deterministic?
Data split	[42]	Yes
Weight initialisation	[Seeds 0-4, one per run]	Yes
Data loader shuffling	[Seeded generator]	Yes
GPU kernels	Example: `torch.use_deterministic_algorithms(True)` not used for speed	No; differences below 0.1 points observed

6. Hyperparameters

Hyperparameter	Search range	Final value	Selection method
Learning rate	[1e-5 to 1e-3, log scale]	[3e-4]	Example: Best mean validation accuracy over 3 seeds
Batch size	[32, 64, 128]	[64]	[Method]
[Hyperparameter]	[Range]	[Value]	[Method]

Search budget: [number of trials, method such as random search or Bayesian optimisation].

7. Compute budget

Item	Value
Hardware	[8 x GPU model, 80 GB]
Time per main run	[6 GPU hours]
Number of runs (final)	[5 seeds x 4 configurations]
Total including tuning	Example: about 1,200 GPU hours
Estimated energy or emissions	[Value and method, if reported]

8. Statistical reporting

Report the mean and spread over $k$ independent runs:

\bar{x} = \frac{1}{k}\sum_{i=1}^{k} x_i, \qquad s = \sqrt{\frac{1}{k-1}\sum_{i=1}^{k}\left(x_i - \bar{x}\right)^2}, \qquad \text{95\% CI} \approx \bar{x} \pm t_{k-1,\,0.975}\,\frac{s}{\sqrt{k}}

Item	Answer
Number of runs per configuration	[5]
What error bars show	[Standard deviation / standard error / 95% CI]
Significance testing	[Paired bootstrap / permutation test / none]
Multiple comparisons	[Correction method or "not applicable"]

9. Evaluation details

Item	Detail
Metrics	Example: Macro-averaged F1 computed with scikit-learn 1.5.1
Test set	[Name, size, version; never used for tuning]
Baselines	[How obtained: rerun with our code / numbers copied from paper]
Preprocessing at test time	[Same as training / differences]
Post-processing	[Thresholds, calibration]

10. Limitations

[Results shown only on [languages / domains / data sizes]]
[Sensitivity to [hyperparameter or seed] observed]
[Parts that could not be made deterministic]
[Known differences between released code and the paper's runs]

11. Verification record

Date	Verifier	Result rerun	Outcome	Notes
YYYY-MM-DD	[@name]	Example: Table 2, row 3	Reproduced within 0.2 points of reported mean	Used released container

Standard and licence note

Based on common practice for computational reproducibility in research software and machine learning, with topics similar to conference reproducibility checklists. This template is original text written for this collection. Template licence: MIT.

Схожі шаблони

Наукові дослідження