Model Training Plan: [Model name]

Plan for a model training run, written and approved before the main run starts: the objective and success criteria, the data and its splits, the architecture and the baseline it must beat, the compute budget, the ablations that de-risk the run, the checkpoint and stop rules, the evaluation gates, and the risks. The compute estimate uses the widely cited approximation of about 6 FLOPs per parameter per training token (Kaplan et al. 2020) and the compute-optimal sizing idea of Hoffmann et al. 2022 as planning concepts, not as guarantees. Replace every [bracketed] value and every row marked Example. Worked examples use a 1-billion-parameter decoder-only language model trained from scratch for code completion.

Document control

Approvals

Revision history

Contents

1. Objective and success criteria
2. Data and splits
3. Baseline
4. Architecture and training configuration
5. Compute budget
6. Ablations and pilot runs
7. Checkpoints and monitoring
8. Stop and restart criteria
9. Evaluation gates
10. Risks and mitigations
11. Schedule and ownership
12. Glossary and references

1. Objective and success criteria

1.1 Objective

[One paragraph: what the model is for, who uses it, and why a new training run is needed rather than an existing model, prompt changes or fine-tuning.]

Example: Train a 1B-parameter code completion model that runs on a single inference accelerator inside the latency budget of the editor plug-in, and that beats the current 350M-parameter production model on completion acceptance without regressing on licence and secret leakage checks.

1.2 Success criteria

1.3 Non-goals

• Example: Natural-language chat quality; this model completes code only.
• Example: Languages outside the top 12 in the data mix.
• [Non-goal]

2. Data and splits

Document each dataset in a Datasheet (ai-data/datasheet-for-datasets) and reference it here. Record the tokenizer version, because token counts change when the tokenizer changes.

2.1 Sources and mixture

2.2 Processing pipeline

2.3 Splits and decontamination

Check decontamination by n-gram overlap between every evaluation set and the training corpus, and record the overlap rate per set. Split by repository, not by file, so that near-identical files from one project cannot sit on both sides of the split.

3. Baseline

If a public model of similar size already meets the success criteria, stop and record the decision instead of training.

4. Architecture and training configuration

4.1 Architecture

4.2 Optimisation

5. Compute budget

5.1 Training FLOPs

For a dense transformer, the training compute is commonly approximated as about 6 floating-point operations per parameter per training token: roughly 2 for the forward pass and 4 for the backward pass. Kaplan et al. (2020) use this approximation and it ignores the attention term that depends on context length, so add a margin for long contexts.

where $N$ is the number of non-embedding parameters and $D$ is the number of training tokens.

Example: $N = 1.0 \times 10^{9}$ and $D = 2.0 \times 10^{10}$, so $C_{\text{train}} \approx 6 \times 10^{9} \times 2 \times 10^{10} = 1.2 \times 10^{20}$ FLOPs.

5.2 Model and data size check

Hoffmann et al. (2022) found that, for a fixed compute budget, the number of parameters and the number of training tokens should grow in roughly equal proportion, and that many large models of the time were trained on too few tokens. A common reading of their results is a ratio of the order of 20 training tokens per parameter for compute-optimal pre-training. Treat this as a sanity check, not a rule: a model meant for cheap inference is often trained on far more tokens than the compute-optimal amount, because inference cost depends on $N$, not on $D$.

Example: $D / N = 2.0 \times 10^{10} / 1.0 \times 10^{9} = 20$ tokens per parameter, close to the compute-optimal ratio. The team may choose to extend $D$ later if ablation ABL-003 shows that loss is still falling steeply.

5.3 Accelerator hours and cost

where $P_{\text{peak}}$ is the accelerator's peak throughput at the training precision and MFU is the model FLOPs utilisation measured in the pilot run.

Each ablation uses a 100M-parameter model on 2B tokens, so it costs $6 \times 10^{8} \times 2 \times 10^{9} = 1.2 \times 10^{18}$ FLOPs, which is 1% of the main run.

5.4 Memory check

A common rule of thumb for mixed-precision training with Adam is about 16 bytes of accelerator memory per parameter for weights, gradients and optimiser state, before activations. Example: 1.0B parameters x 16 bytes = 16 GB of state, sharded across 16 accelerators, which leaves room for activations at the chosen micro-batch. Confirm the peak memory in the pilot run.

6. Ablations and pilot runs

Run ablations with at least two seeds when the expected effect is small, and report the spread. Record each run in an Experiment Tracking Report (ai-data/experiment-tracking-report).

7. Checkpoints and monitoring

7.1 Checkpoint policy

7.2 Signals and alerts

8. Stop and restart criteria

9. Evaluation gates

Use the method in the LLM Evaluation Plan (ai-data/llm-evaluation-plan) or the Benchmark Report (ai-data/benchmark-report) for confidence intervals and significance. Release documentation goes into a Model Card (ai-data/model-card).

10. Risks and mitigations

11. Schedule and ownership

After the run, record the actual compute, MFU, number of restarts and final scores next to the planned values. The difference between plan and actual is the most useful input to the next plan.

12. Glossary and references

References:

• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. Source of the approximation of about 6N training FLOPs per token (section 2.1 of the paper).
• Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. Source of the finding that parameters and tokens should scale in roughly equal proportion.
• Pineau, J. et al. (2021). Improving Reproducibility in Machine Learning Research. arXiv:2003.12206 (JMLR).
• Related templates: ML System Design (ai-data/ml-system-design), Fine-Tuning Report (ai-data/fine-tuning-report), Datasheet for Datasets (ai-data/datasheet-for-datasets), Experiment Tracking Report (ai-data/experiment-tracking-report), Model Card (ai-data/model-card), Reproducibility Checklist (science/reproducibility-checklist).

Standard and licence note

Based on common machine learning engineering practice and on the published research cited above (Kaplan et al. 2020; Hoffmann et al. 2022), used here as planning concepts only. No formal standard governs training plans. The text, tables and worked example are original; example figures are illustrative and must be replaced with measured values. Check the current versions of the cited papers and your hardware vendor's datasheet before relying on any number. Template licence: MIT.