Statistical Analysis Plan Template (Statistics, Thesis or Project)

Statistical Analysis Plan: [Your thesis or project title]

A statistical analysis plan (SAP) for a thesis, a dissertation chapter or an applied project in any field. It fixes, before you see the outcome data, what you will estimate, which hypotheses are confirmatory, which model you will fit, how you will check its assumptions and what you will do if they fail, how you will handle missing data and multiple tests, and how you will log every later change. Writing the plan in this order teaches the decisions a statistician makes, and the filled sections give you a model answer to copy the shape of. The worked examples use fictional data; replace them with your own. Source basis: the ICH E9(R1) estimand framework, Gamble et al. (2017) on SAP content, and standard methods texts, written in original words.

Document control

This table identifies the plan, its owner and the software it commits to.

When to write a plan and what it protects

A statistical analysis plan exists because data analysis contains many small, defensible choices: which covariates, which transformation, which outliers to drop, which subgroup to look at. Made after seeing the data, these choices drift (often without anyone intending it) towards the more interesting result. Gelman and Loken (2014) called this the "garden of forking paths". A plan made in advance closes most of those paths, so that the p-values and confidence intervals in the confirmatory analysis mean what they claim to mean.

Write the plan when the design and the measures are fixed and before you can see outcomes by group. In a thesis, the natural moment is just after ethics approval and before data collection ends. A plan does not forbid exploration: it labels exploration honestly, so a reader can weigh it. For clinical readers, the plan below follows the vocabulary of the ICH E9(R1) addendum on estimands and sensitivity analysis; for non-clinical readers the same ideas work with plain words.

1. Objectives and estimands

State the objective in one sentence, then define the estimand: the exact quantity the study is meant to estimate. The ICH E9(R1) framework describes an estimand by five attributes: the population, the treatment conditions compared, the outcome variable, the handling of intercurrent events (things that happen after assignment and affect the interpretation of the outcome, such as stopping the intervention), and the population-level summary.

Example (Lakemoor University Calm Campus trial). Objective: to estimate the effect of a four-week mindfulness app programme, compared with a waitlist, on perceived stress during the exam period among Lakemoor University undergraduates.

A treatment policy strategy answers "what happens if we offer the app", which is the policy question a university cares about. A hypothetical strategy ("what if everyone had used it as intended") answers a different question and needs stronger assumptions. Choose the question first; the strategy follows.

2. Hypotheses

List hypotheses in three tiers. Primary: the one hypothesis that the sample size is built for and that decides the study's main conclusion. Secondary: a short list of supporting confirmatory hypotheses, protected by a multiplicity rule. Exploratory: everything else, reported with estimates and intervals and labelled as hypothesis-generating.

With $\Delta$ the adjusted mean difference in week-4 PSS-10 (app minus waitlist), the primary hypothesis pair is

Two-sided tests are the default even when you expect a decrease: a one-sided test must be justified in advance and cannot be chosen after seeing the direction of the result.

3. Design and data

Describe the unit of analysis, how units were selected and allocated, the timing of measurements, and any blinding. The model must respect the design: stratification and clustering in the design appear in the model.

Example. Two-arm, parallel-group randomised trial with individual allocation 1:1, stratified by year of study (first year vs later years) using permuted blocks of four, generated with a recorded seed by a staff member outside the research team. Measures at baseline (week 0) and week 4, online. The participant cannot be blinded to the arm; the analyst works on a data set with arms coded as X and Y until the primary analysis code is frozen. Data are stored on the Lakemoor University research drive under the approved data management plan. See academic/stats-experiment-design for randomisation principles and academic/stats-sampling-plan for recruitment.

4. Variables

List each variable with its role in the analysis, measurement level and coding. The full codebook (labels, valid ranges, missing-value codes, derivation of scale scores) lives in academic/stats-variable-codebook; the plan points to it so the two never disagree.

5. Sample size justification

A sample size justification states the target effect, the variability, α, the power, the analysis it refers to, and the expected loss. The smallest effect worth detecting should come from what matters in practice (a minimal important difference) or from a pilot, not from what a previous study happened to find.

For a two-sample comparison of means with standardised effect $d$, the approximate size per group is

and adjusting for a baseline covariate with correlation $\rho$ to the outcome multiplies it by about $(1 - \rho^2)$ (Borm, Fransen and Lemmens 2007).

Example. Target $d = 0.5$ (3 PSS-10 points with an assumed SD of 6 from the department's pilot), α = 0.05 two-sided, power 0.80, $\rho = 0.5$, attrition 15%.

library(pwr)   # install.packages("pwr") once
p <- pwr.t.test(d = 0.5, power = 0.80, sig.level = 0.05, type = "two.sample")
n_t <- ceiling(p$n)
n_ancova <- ceiling(n_t * (1 - 0.5^2))
n_final <- ceiling(n_ancova / (1 - 0.15))
cat("t-test n/group:", round(p$n, 2), "->", n_t, "\n")
cat("ANCOVA n/group:", n_ancova, " after attrition:", n_final, " total:", 2 * n_final, "\n")

t-test n/group: 63.77 -> 64
ANCOVA n/group: 48  after attrition: 57  total: 114

import numpy as np
from statsmodels.stats.power import TTestIndPower

n_t = TTestIndPower().solve_power(effect_size=0.5, power=0.80, alpha=0.05)
n_anc = int(np.ceil(np.ceil(n_t) * (1 - 0.5**2)))
n_fin = int(np.ceil(n_anc / 0.85))
print(f"t-test n/group: {n_t:.2f}  ANCOVA: {n_anc}  after attrition: {n_fin}  total: {2*n_fin}")

t-test n/group: 63.77  ANCOVA: 48  after attrition: 57  total: 114

Both tools agree: recruit 57 per arm, 114 in total. The power curve shows why the target effect matters so much: at 60 per group, power is 0.775 for $d = 0.5$ but only 0.584 for $d = 0.4$ (values from pwr.t.test, unadjusted t-test).

6. Analysis populations

Define who is included in each analysis, by rules decided now. The full analysis set (often called intention to treat) includes everyone randomised, analysed in the arm they were assigned to; it preserves the benefit of randomisation and matches the treatment policy estimand. A per-protocol set keeps only those who followed the protocol; it answers a different question and is prone to bias, so it belongs in sensitivity analyses.

7. Primary analysis model

The primary model must estimate the primary estimand directly. For a randomised two-arm trial with a baseline measurement, analysis of covariance (ANCOVA) is the standard choice: it is unbiased under randomisation and usually more precise than comparing change scores.

Report $\hat{\beta}_1$ with its 95% CI and two-sided p-value, plus a standardised difference ($\hat{\beta}_1$ divided by the pooled baseline SD). The dry run below uses 16 invented rows (no stratum column, to keep it short) to prove that the code runs and the output columns are where the report template expects them. These numbers are not results.

dummy <- data.frame(
  arm  = factor(rep(c("waitlist", "app"), 8), levels = c("waitlist", "app")),
  pss0 = c(22, 18, 25, 20, 17, 24, 21, 19, 26, 23, 16, 20, 24, 22, 18, 21),
  pss4 = c(21, 15, 24, 17, 17, 20, 20, 16, 25, 19, 15, 18, 22, 18, 18, 17))
fit <- lm(pss4 ~ arm + pss0, data = dummy)
round(coef(summary(fit)), 3)
round(confint(fit)["armapp", ], 3)

            Estimate Std. Error t value Pr(>|t|)
(Intercept)    1.974      1.097   1.800    0.095
armapp        -2.534      0.291  -8.712    0.000
pss0           0.865      0.051  16.958    0.000
 2.5 % 97.5 %
-3.162 -1.905

import pandas as pd
import statsmodels.formula.api as smf

dummy = pd.DataFrame({
    "arm": ["waitlist", "app"] * 8,
    "pss0": [22, 18, 25, 20, 17, 24, 21, 19, 26, 23, 16, 20, 24, 22, 18, 21],
    "pss4": [21, 15, 24, 17, 17, 20, 20, 16, 25, 19, 15, 18, 22, 18, 18, 17]})
fit = smf.ols("pss4 ~ C(arm, Treatment('waitlist')) + pss0", data=dummy).fit()
print(fit.summary2().tables[1].iloc[:, [0, 1, 4, 5]].round(3).to_string())

                                      Coef.  Std.Err.  [0.025  0.975]
Intercept                             1.974     1.097  -0.396   4.344
C(arm, Treatment('waitlist'))[T.app] -2.534     0.291  -3.162  -1.905
pss0                                  0.865     0.051   0.755   0.975

Note the reference category: R orders factor levels alphabetically unless you set levels, and statsmodels does the same unless you write Treatment('waitlist'). Without that, the sign of the estimate flips between tools. Fix the reference level in the plan.

8. Assumption checks and fallbacks decided in advance

Every model rests on assumptions. The plan states how each will be checked and what you will do if it fails, so the choice of fallback cannot depend on the result. Prefer graphical checks plus a pre-set rule over a cascade of formal tests: in a sample of 114, normality tests flag trivial departures, and in a sample of 20 they miss serious ones. Full diagnostics per method family are in academic/stats-assumption-checklist.

9. Missing data strategy

Missing outcomes are the main threat to a randomised comparison, because the people who drop out may differ by arm. State the expected amount, the assumed mechanism and the method. Data are MCAR (missing completely at random) if missingness is unrelated to anything; MAR (missing at random) if it depends only on observed variables; MNAR (missing not at random) if it depends on the unobserved value itself. You cannot test MAR against MNAR from the data, which is why a sensitivity analysis is required. Record actual missingness in academic/stats-data-cleaning-log.

Example. Expected week-4 missingness 15%. Primary: multiple imputation under MAR (Rubin 1987), 20 imputed data sets, imputation model including arm, year, baseline PSS-10, baseline PSQI and WEMWBS, imputed separately by arm, results combined with Rubin's rules (R package mice, or statsmodels.imputation.mice in Python). Sensitivity S1: complete-case ANCOVA. Sensitivity S2: delta-adjusted tipping point, adding δ = 1, 2, 3 and 4 PSS-10 points to imputed values in the app arm only, to find how pessimistic the missing app outcomes must be to remove the effect. Single imputation by last observation carried forward will not be used.

10. Multiplicity

Each additional confirmatory test raises the chance of at least one false positive. Decide the family and the method now. The Holm procedure (Holm 1979) controls the family-wise error rate and is uniformly more powerful than Bonferroni: sort the $m$ p-values, compare the smallest with $\alpha/m$, the next with $\alpha/(m-1)$, and stop at the first that fails. Benjamini-Hochberg controls the false discovery rate instead and suits large exploratory families. See academic/stats-hypothesis-testing.

Example. Primary hypothesis at α = 0.05. Secondary family {PSQI, WEMWBS} tested with Holm at α = 0.05, only if the primary test is significant (a gatekeeping order). Exploratory analyses are not corrected and are labelled exploratory. The dry run with invented p-values of 0.021 and 0.048:

p.adjust(c(sleep = 0.021, wellbeing = 0.048), method = "holm")

    sleep wellbeing
    0.042     0.048

from statsmodels.stats.multitest import multipletests
print(multipletests([0.021, 0.048], method="holm")[1])

[0.042 0.048]

Both adjusted values are below 0.05, so both would be declared significant; with Bonferroni the second (0.096) would not.

11. Subgroup and sensitivity analyses

A subgroup analysis asks whether the effect differs between groups; test it with an interaction term and report the difference in effects with its CI. Comparing "significant in group A, not significant in group B" is a classic error: the difference between significant and not significant is not itself significant. A sensitivity analysis asks whether the main conclusion survives a change in one assumption.

12. Deviations log

Record each departure from the approved plan: what changed, why, when, whether outcome data had been seen, and the impact. Report the log in the thesis methods or an appendix.

13. Timeline

Worked example B: a secondary hypothesis (half solved)

Write the plan entry for secondary hypothesis 1 (sleep quality, PSQI 0 to 21, higher is worse).

1. Estimand variable: PSQI global score at week [ ].
2. Intercurrent event "stopped using the app": strategy = [ ], because [ ].
3. Hypotheses: $H_0: \Delta_{\text{PSQI}} = 0$ vs $H_1$: [ ].
4. Model: lm(psqi4 ~ arm + [ ] + year), reference arm = [ ].
5. Multiplicity: tested with [ ] together with WEMWBS, only if [ ].
6. Missing data: same as the primary, multiple imputation under [ ] with tipping-point sensitivity.

Exercise C: plan a different study

The Lakemoor University library wants to know whether extending opening hours to midnight changes the weekly number of study-room bookings. Six branch libraries are available; three will be chosen at random to extend hours for eight weeks. Bookings are counted weekly for eight weeks before and eight weeks during the change. Write: (1) the estimand with five attributes, (2) the unit of randomisation and why it matters for the analysis, (3) a primary model, (4) one sensitivity analysis.

Common mistakes and misconceptions

• "Appropriate tests will be used." Not a plan. Name the model, the covariates, the reference level and the summary.
• Choosing the fallback after seeing results. "We switched to Mann-Whitney because the t-test was not significant" is a forking path. Decide the rule in section 8.
• Analysing change scores and ANCOVA, then reporting the better one. Pick one as primary in advance; ANCOVA is usually more precise in randomised designs.
• Testing baseline balance in a randomised trial. Baseline differences in a randomised study are chance by definition; a p-value for them answers no useful question. Describe baseline, do not test it.
• Subgroup claims from separate p-values. Use an interaction and its CI.
• Treating per-protocol as the "true" effect. It conditions on post-randomisation behaviour and can be badly biased.
• Sample size from a published effect. Published effects are inflated by selection. Use a minimal important difference or a conservative pilot estimate.
• p = 0.03 means a 3% chance that H0 is true. No: it is the probability of data at least this extreme if H0 and the model were true (Wasserstein and Lazar 2016).

Self-test questions

1. Name the five attributes of an estimand in the ICH E9(R1) framework.
2. Why should the fallback for a failed assumption be written before the data are seen?
3. A plan needs 64 per group for an unadjusted t-test. With a baseline covariate correlated 0.6 with the outcome, about how many per group does ANCOVA need, before attrition?
4. Three secondary p-values are 0.010, 0.030 and 0.040. Which are significant under Holm at α = 0.05?
5. What is the difference between a sensitivity analysis and a subgroup analysis?

My notes and next steps

• Confidence (1 to 5): estimands [ ], sample size [ ], missing data [ ], multiplicity [ ].
• Open questions for my supervisor: [ ]
• Mistakes to log in academic/stats-error-log: [ ]
• Next: write the codebook (academic/stats-variable-codebook), then the report shell (academic/stats-analysis-report).

Sources and further reading

Structure aligned with the estimand attributes of ICH E9(R1) and the SAP content items of Gamble et al. (2017), adapted for theses and projects outside clinical trials and written in original words.

• ICH (2019). E9(R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use.
• Gamble, C., Krishan, A., Stocken, D., Lewis, S., Juszczak, E., Doré, C., et al. (2017). Guidelines for the content of statistical analysis plans in clinical trials. JAMA, 318(23), 2337–2343.
• Borm, G. F., Fransen, J., and Lemmens, W. A. J. G. (2007). A simple sample size formula for analysis of covariance in randomized clinical trials. Journal of Clinical Epidemiology, 60(12), 1234–1238.
• Gelman, A., and Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460–465.
• Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
• Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1), 289–300.
• Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley.
• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates.
• Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133.
• Further reading (by name only): de Smith, M. J. (2024). Statistical Analysis Handbook (2024 edition, StatsRef). See also academic/pre-analysis-plan (economics) and science/preregistration (OSF-style form).

Answer key

Example B. (1) Week 4. (2) Treatment policy, because the question is the effect of offering the app, whatever use follows. (3) $H_1: \Delta_{\text{PSQI}} \neq 0$ (two-sided). (4) lm(psqi4 ~ arm + psqi0 + year), reference waitlist. (5) Holm, together with WEMWBS, only if the primary hypothesis is significant (gatekeeping). (6) MAR.

Exercise C (one good answer). (1) Population: the six branch libraries and their weekly bookings; treatment: midnight closing vs usual hours; variable: weekly bookings during the eight change weeks; intercurrent event: a branch closed for repairs, handled by treatment policy (or excluded weeks recorded as a deviation); summary: ratio of mean weekly bookings, extended vs usual, adjusted for the pre-period. (2) The branch is the unit of randomisation, so the effective sample size is six, not the number of weeks; analysing weeks as independent is pseudo-replication. (3) A Poisson or negative binomial model of weekly counts with branch random intercepts (or a branch-level ANCOVA on during-period means with the pre-period mean as covariate), reporting the rate ratio with a 95% CI; see academic/stats-count-regression. (4) A permutation test over the 20 possible assignments of three of six branches. Note its limit: the smallest attainable one-sided p-value is 1/20 = 0.05, so state in the plan that six branches give an estimate with an interval rather than a decisive test.

Self-test. (1) Population, treatment conditions, variable, handling of intercurrent events, population-level summary. (2) So the choice cannot depend on which option gives the preferred result. (3) $64 \times (1 - 0.36) = 40.96$, about 41 per group. (4) Holm: 0.010 vs 0.05/3 = 0.0167, significant; 0.030 vs 0.05/2 = 0.025, not significant; stop. Only the first. (5) A sensitivity analysis checks whether the main conclusion survives a changed assumption; a subgroup analysis asks whether the effect itself differs between groups.

Revision history