# Statistical Analysis Plan: [Your thesis or project title] > A statistical analysis plan (SAP) for a thesis, a dissertation chapter or an applied project in any field. It fixes, > before you see the outcome data, what you will estimate, which hypotheses are confirmatory, which model you will fit, > how you will check its assumptions and what you will do if they fail, how you will handle missing data and multiple > tests, and how you will log every later change. Writing the plan in this order teaches the decisions a statistician > makes, and the filled sections give you a model answer to copy the shape of. The worked examples use fictional > data; replace them with your own. Source basis: the ICH E9(R1) estimand framework, Gamble et al. (2017) on SAP > content, and standard methods texts, written in original words. ## Document control This table identifies the plan, its owner and the software it commits to. | Field | Value | |---|---| | Course or project | [MSc thesis / PhD chapter / client project] | | Learner or analyst | [Your name], supervisor [Supervisor name] | | Data set | [*Example:* Lakemoor University Calm Campus trial (fictional), planned n = 114] | | Software and versions | R 4.5.1 with pwr 1.3.0; Python 3.14, numpy 2.5.4, pandas 3.1.0, statsmodels 0.15.0 | | Registration | [OSF Registries DOI or "not registered"] | | Date | YYYY-MM-DD | | Version | [1.0] | | Status | [Draft / Approved before data lock / Amended] | ## When to write a plan and what it protects A statistical analysis plan exists because data analysis contains many small, defensible choices: which covariates, which transformation, which outliers to drop, which subgroup to look at. Made after seeing the data, these choices drift (often without anyone intending it) towards the more interesting result. Gelman and Loken (2014) called this the "garden of forking paths". A plan made in advance closes most of those paths, so that the p-values and confidence intervals in the confirmatory analysis mean what they claim to mean. Write the plan when the design and the measures are fixed and before you can see outcomes by group. In a thesis, the natural moment is just after ethics approval and before data collection ends. A plan does not forbid exploration: it labels exploration honestly, so a reader can weigh it. For clinical readers, the plan below follows the vocabulary of the ICH E9(R1) addendum on estimands and sensitivity analysis; for non-clinical readers the same ideas work with plain words. ```mermaid flowchart TD Q["Research question"] --> E["Estimand: what exactly is estimated"] E --> H["Hypotheses: primary, secondary, exploratory"] H --> D["Design, data and variables"] D --> N["Sample size justification"] N --> M["Primary model with formula"] M --> A["Assumption checks and pre-set fallbacks"] A --> MD["Missing data strategy"] MD --> MT["Multiplicity rule"] MT --> S["Subgroup and sensitivity analyses"] S --> L["Approve, date, version"] L --> DV["Deviations log during the study"] ``` ## 1. Objectives and estimands State the objective in one sentence, then define the **estimand**: the exact quantity the study is meant to estimate. The ICH E9(R1) framework describes an estimand by five attributes: the population, the treatment conditions compared, the outcome variable, the handling of **intercurrent events** (things that happen after assignment and affect the interpretation of the outcome, such as stopping the intervention), and the population-level summary. *Example (Lakemoor University Calm Campus trial).* Objective: to estimate the effect of a four-week mindfulness app programme, compared with a waitlist, on perceived stress during the exam period among Lakemoor University undergraduates. | Attribute | Primary estimand | |---|---| | Population | *Example:* Undergraduates enrolled at Lakemoor University in the autumn term, aged 18 or over, PSS-10 ≥ 14 at screening | | Treatment conditions | *Example:* Access to the app programme for four weeks vs waitlist (access after week 4) | | Variable | *Example:* Perceived Stress Scale (PSS-10) total, 0 to 40, at week 4 | | Intercurrent events and strategy | *Example:* Stopping app use: treatment policy (outcome used regardless). Starting counselling: treatment policy | | Population-level summary | *Example:* Difference in mean week-4 PSS-10 (app minus waitlist), adjusted for baseline PSS-10 and year of study | A **treatment policy** strategy answers "what happens if we offer the app", which is the policy question a university cares about. A **hypothetical** strategy ("what if everyone had used it as intended") answers a different question and needs stronger assumptions. Choose the question first; the strategy follows. ## 2. Hypotheses List hypotheses in three tiers. **Primary**: the one hypothesis that the sample size is built for and that decides the study's main conclusion. **Secondary**: a short list of supporting confirmatory hypotheses, protected by a multiplicity rule. **Exploratory**: everything else, reported with estimates and intervals and labelled as hypothesis-generating. With $\Delta$ the adjusted mean difference in week-4 PSS-10 (app minus waitlist), the primary hypothesis pair is $$ H_0: \Delta = 0 \qquad \text{versus} \qquad H_1: \Delta \neq 0, \qquad \alpha = 0.05 \text{ two-sided}. $$ | Tier | Hypothesis | Analysis | Section | |---|---|---|---| | *Example:* Primary | App lowers week-4 PSS-10 vs waitlist | ANCOVA, adjusted difference with 95% CI | 7 | | *Example:* Secondary 1 | App improves sleep quality (PSQI, week 4) | ANCOVA, Holm-adjusted | 10 | | *Example:* Secondary 2 | App improves well-being (WEMWBS, week 4) | ANCOVA, Holm-adjusted | 10 | | *Example:* Exploratory | Effect differs by year of study | Interaction term, estimate only | 11 | Two-sided tests are the default even when you expect a decrease: a one-sided test must be justified in advance and cannot be chosen after seeing the direction of the result. ## 3. Design and data Describe the unit of analysis, how units were selected and allocated, the timing of measurements, and any blinding. The model must respect the design: stratification and clustering in the design appear in the model. *Example.* Two-arm, parallel-group randomised trial with individual allocation 1:1, stratified by year of study (first year vs later years) using permuted blocks of four, generated with a recorded seed by a staff member outside the research team. Measures at baseline (week 0) and week 4, online. The participant cannot be blinded to the arm; the analyst works on a data set with arms coded as X and Y until the primary analysis code is frozen. Data are stored on the Lakemoor University research drive under the approved data management plan. See academic/stats-experiment-design for randomisation principles and academic/stats-sampling-plan for recruitment. ## 4. Variables List each variable with its role in the analysis, measurement level and coding. The full codebook (labels, valid ranges, missing-value codes, derivation of scale scores) lives in academic/stats-variable-codebook; the plan points to it so the two never disagree. | Variable | Role | Level | Coding or derivation | |---|---|---|---| | *Example:* `arm` | Treatment | Nominal | waitlist (reference), app | | *Example:* `pss0`, `pss4` | Baseline covariate, primary outcome | Interval (treated as) | Sum of 10 items, 4 reverse-scored, 0 to 40; prorate if exactly one item missing | | *Example:* `year` | Stratification covariate | Binary | first, later | | *Example:* `psqi4`, `wemwbs4` | Secondary outcomes | Interval (treated as) | Published scoring rules | | *Example:* `sessions` | Adherence (descriptive and sensitivity) | Count | App sessions completed, 0 to 28 | ## 5. Sample size justification A sample size justification states the target effect, the variability, α, the power, the analysis it refers to, and the expected loss. The smallest effect worth detecting should come from what matters in practice (a minimal important difference) or from a pilot, not from what a previous study happened to find. For a two-sample comparison of means with standardised effect $d$, the approximate size per group is $$ n \approx \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^2}{d^2}, $$ and adjusting for a baseline covariate with correlation $\rho$ to the outcome multiplies it by about $(1 - \rho^2)$ (Borm, Fransen and Lemmens 2007). *Example.* Target $d = 0.5$ (3 PSS-10 points with an assumed SD of 6 from the department's pilot), α = 0.05 two-sided, power 0.80, $\rho = 0.5$, attrition 15%. ```r library(pwr) # install.packages("pwr") once p <- pwr.t.test(d = 0.5, power = 0.80, sig.level = 0.05, type = "two.sample") n_t <- ceiling(p$n) n_ancova <- ceiling(n_t * (1 - 0.5^2)) n_final <- ceiling(n_ancova / (1 - 0.15)) cat("t-test n/group:", round(p$n, 2), "->", n_t, "\n") cat("ANCOVA n/group:", n_ancova, " after attrition:", n_final, " total:", 2 * n_final, "\n") ``` ```text t-test n/group: 63.77 -> 64 ANCOVA n/group: 48 after attrition: 57 total: 114 ``` ```python import numpy as np from statsmodels.stats.power import TTestIndPower n_t = TTestIndPower().solve_power(effect_size=0.5, power=0.80, alpha=0.05) n_anc = int(np.ceil(np.ceil(n_t) * (1 - 0.5**2))) n_fin = int(np.ceil(n_anc / 0.85)) print(f"t-test n/group: {n_t:.2f} ANCOVA: {n_anc} after attrition: {n_fin} total: {2*n_fin}") ``` ```text t-test n/group: 63.77 ANCOVA: 48 after attrition: 57 total: 114 ``` Both tools agree: recruit 57 per arm, 114 in total. The power curve shows why the target effect matters so much: at 60 per group, power is 0.775 for $d = 0.5$ but only 0.584 for $d = 0.4$ (values from `pwr.t.test`, unadjusted t-test). ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "description": "Power of a two-sided two-sample t-test at alpha 0.05 by sample size per group", "width": 380, "height": 220, "data": {"values": [ {"n": 20, "d": "d = 0.4", "power": 0.234}, {"n": 40, "d": "d = 0.4", "power": 0.424}, {"n": 60, "d": "d = 0.4", "power": 0.584}, {"n": 80, "d": "d = 0.4", "power": 0.710}, {"n": 100, "d": "d = 0.4", "power": 0.804}, {"n": 20, "d": "d = 0.5", "power": 0.338}, {"n": 40, "d": "d = 0.5", "power": 0.598}, {"n": 60, "d": "d = 0.5", "power": 0.775}, {"n": 80, "d": "d = 0.5", "power": 0.882}, {"n": 100, "d": "d = 0.5", "power": 0.940} ]}, "layer": [ { "mark": {"type": "line", "point": true}, "encoding": { "x": {"field": "n", "type": "quantitative", "title": "Sample size per group"}, "y": {"field": "power", "type": "quantitative", "title": "Power", "scale": {"domain": [0, 1]}}, "color": {"field": "d", "type": "nominal", "title": "Effect size"} } }, { "mark": {"type": "rule", "strokeDash": [4, 4]}, "encoding": {"y": {"datum": 0.8}} } ] } ``` ## 6. Analysis populations Define who is included in each analysis, by rules decided now. The **full analysis set** (often called intention to treat) includes everyone randomised, analysed in the arm they were assigned to; it preserves the benefit of randomisation and matches the treatment policy estimand. A **per-protocol set** keeps only those who followed the protocol; it answers a different question and is prone to bias, so it belongs in sensitivity analyses. | Analysis set | Rule | Used for | |---|---|---| | *Example:* Full analysis set | All randomised participants, by assigned arm | Primary and secondary analyses | | *Example:* Per-protocol set | Completed baseline and week 4; app arm used the app in at least 3 of 4 weeks; no protocol violation | Sensitivity analysis S3 | | *Example:* Safety set | Everyone who opened the app at least once | Reporting of distress alerts | ## 7. Primary analysis model The primary model must estimate the primary estimand directly. For a randomised two-arm trial with a baseline measurement, analysis of covariance (ANCOVA) is the standard choice: it is unbiased under randomisation and usually more precise than comparing change scores. $$ \text{PSS4}_i = \beta_0 + \beta_1\,\text{App}_i + \beta_2\,\text{PSS0}_i + \beta_3\,\text{Later}_i + \varepsilon_i, \qquad \hat{\Delta} = \hat{\beta}_1 . $$ | Symbol | Meaning | |---|---| | $\text{App}_i$ | 1 if participant $i$ was assigned to the app, 0 for waitlist (reference) | | $\text{PSS0}_i$ | baseline PSS-10, centred at the sample mean (the dry run below leaves it uncentred, which changes only the intercept) | | $\text{Later}_i$ | 1 if in second year or later (stratification factor) | | $\beta_1$ | adjusted mean difference in week-4 PSS-10, the primary estimate | | $\varepsilon_i$ | error, mean 0, constant variance $\sigma^2$ | Report $\hat{\beta}_1$ with its 95% CI and two-sided p-value, plus a standardised difference ($\hat{\beta}_1$ divided by the pooled baseline SD). The dry run below uses 16 **invented** rows (no stratum column, to keep it short) to prove that the code runs and the output columns are where the report template expects them. These numbers are not results. ```r dummy <- data.frame( arm = factor(rep(c("waitlist", "app"), 8), levels = c("waitlist", "app")), pss0 = c(22, 18, 25, 20, 17, 24, 21, 19, 26, 23, 16, 20, 24, 22, 18, 21), pss4 = c(21, 15, 24, 17, 17, 20, 20, 16, 25, 19, 15, 18, 22, 18, 18, 17)) fit <- lm(pss4 ~ arm + pss0, data = dummy) round(coef(summary(fit)), 3) round(confint(fit)["armapp", ], 3) ``` ```text Estimate Std. Error t value Pr(>|t|) (Intercept) 1.974 1.097 1.800 0.095 armapp -2.534 0.291 -8.712 0.000 pss0 0.865 0.051 16.958 0.000 2.5 % 97.5 % -3.162 -1.905 ``` ```python import pandas as pd import statsmodels.formula.api as smf dummy = pd.DataFrame({ "arm": ["waitlist", "app"] * 8, "pss0": [22, 18, 25, 20, 17, 24, 21, 19, 26, 23, 16, 20, 24, 22, 18, 21], "pss4": [21, 15, 24, 17, 17, 20, 20, 16, 25, 19, 15, 18, 22, 18, 18, 17]}) fit = smf.ols("pss4 ~ C(arm, Treatment('waitlist')) + pss0", data=dummy).fit() print(fit.summary2().tables[1].iloc[:, [0, 1, 4, 5]].round(3).to_string()) ``` ```text Coef. Std.Err. [0.025 0.975] Intercept 1.974 1.097 -0.396 4.344 C(arm, Treatment('waitlist'))[T.app] -2.534 0.291 -3.162 -1.905 pss0 0.865 0.051 0.755 0.975 ``` Note the reference category: R orders factor levels alphabetically unless you set `levels`, and statsmodels does the same unless you write `Treatment('waitlist')`. Without that, the sign of the estimate flips between tools. Fix the reference level in the plan. ## 8. Assumption checks and fallbacks decided in advance Every model rests on assumptions. The plan states how each will be checked and what you will do if it fails, so the choice of fallback cannot depend on the result. Prefer graphical checks plus a pre-set rule over a cascade of formal tests: in a sample of 114, normality tests flag trivial departures, and in a sample of 20 they miss serious ones. Full diagnostics per method family are in academic/stats-assumption-checklist. | Assumption | Check | Pre-set action if violated | |---|---|---| | *Example:* Independence | Design: individual randomisation, one record per person | Not expected; if duplicates found, keep first record and log | | *Example:* Linearity of PSS0 | Residuals vs baseline plot with a smoother | Add a natural spline of PSS0 (3 df) | | *Example:* Equal slopes across arms | Fit `arm * pss0`, inspect, do not test for selection | Report as sensitivity S4; primary model unchanged | | *Example:* Constant variance | Residuals vs fitted; spread ratio of arms > 2 | Report HC3 robust SE as primary inference | | *Example:* Approximately normal residuals | Q-Q plot | With n = 114, rely on CLT; add bootstrap CI (2,000 resamples, seed 2026) as sensitivity | | *Example:* Influential points | Cook's distance > 4/n | Keep all data in primary; refit without them as sensitivity S5 | ```mermaid flowchart TD F["Fit primary ANCOVA"] --> R["Residual plots and Cook's distance"] R --> V{"Spread ratio of arms above 2?"} V -->|Yes| HC["Primary inference uses HC3 robust SE"] V -->|No| OLS["Primary inference uses model SE"] HC --> L{"Curvature in PSS0?"} OLS --> L L -->|Yes| SP["Add natural spline of PSS0"] L -->|No| KEEP["Keep linear term"] SP --> REP["Report and log the rule that fired"] KEEP --> REP ``` ## 9. Missing data strategy Missing outcomes are the main threat to a randomised comparison, because the people who drop out may differ by arm. State the expected amount, the assumed mechanism and the method. Data are **MCAR** (missing completely at random) if missingness is unrelated to anything; **MAR** (missing at random) if it depends only on observed variables; **MNAR** (missing not at random) if it depends on the unobserved value itself. You cannot test MAR against MNAR from the data, which is why a sensitivity analysis is required. Record actual missingness in academic/stats-data-cleaning-log. *Example.* Expected week-4 missingness 15%. Primary: multiple imputation under MAR (Rubin 1987), 20 imputed data sets, imputation model including arm, year, baseline PSS-10, baseline PSQI and WEMWBS, imputed separately by arm, results combined with Rubin's rules (R package `mice`, or `statsmodels.imputation.mice` in Python). Sensitivity S1: complete-case ANCOVA. Sensitivity S2: delta-adjusted tipping point, adding δ = 1, 2, 3 and 4 PSS-10 points to imputed values in the app arm only, to find how pessimistic the missing app outcomes must be to remove the effect. Single imputation by last observation carried forward will not be used. ## 10. Multiplicity Each additional confirmatory test raises the chance of at least one false positive. Decide the family and the method now. The **Holm** procedure (Holm 1979) controls the family-wise error rate and is uniformly more powerful than Bonferroni: sort the $m$ p-values, compare the smallest with $\alpha/m$, the next with $\alpha/(m-1)$, and stop at the first that fails. **Benjamini-Hochberg** controls the false discovery rate instead and suits large exploratory families. See academic/stats-hypothesis-testing. *Example.* Primary hypothesis at α = 0.05. Secondary family {PSQI, WEMWBS} tested with Holm at α = 0.05, only if the primary test is significant (a gatekeeping order). Exploratory analyses are not corrected and are labelled exploratory. The dry run with invented p-values of 0.021 and 0.048: ```r p.adjust(c(sleep = 0.021, wellbeing = 0.048), method = "holm") ``` ```text sleep wellbeing 0.042 0.048 ``` ```python from statsmodels.stats.multitest import multipletests print(multipletests([0.021, 0.048], method="holm")[1]) ``` ```text [0.042 0.048] ``` Both adjusted values are below 0.05, so both would be declared significant; with Bonferroni the second (0.096) would not. ## 11. Subgroup and sensitivity analyses A **subgroup analysis** asks whether the effect differs between groups; test it with an interaction term and report the difference in effects with its CI. Comparing "significant in group A, not significant in group B" is a classic error: the difference between significant and not significant is not itself significant. A **sensitivity analysis** asks whether the main conclusion survives a change in one assumption. | ID | Analysis | Assumption varied | |---|---|---| | *Example:* S1 | Complete-case ANCOVA | MAR imputation | | *Example:* S2 | Delta-adjusted tipping point | MAR vs MNAR | | *Example:* S3 | Per-protocol set | Adherence (answers a different question) | | *Example:* S4 | Add arm by baseline interaction | Equal slopes | | *Example:* S5 | Exclude influential points | Influence | | *Example:* G1 (subgroup) | `arm * year`, report interaction estimate and CI | Effect homogeneity by year | ## 12. Deviations log Record each departure from the approved plan: what changed, why, when, whether outcome data had been seen, and the impact. Report the log in the thesis methods or an appendix. | Date | Section | Change | Reason | Outcome data seen? | Impact | |---|---|---|---|---|---| | *Example:* YYYY-MM-DD | 4 | WEMWBS short form (7 items) instead of 14 | Survey length limit set by the ethics committee | No | Secondary outcome scale changed; noted in report | | [YYYY-MM-DD] | [ ] | [ ] | [ ] | [Yes / No] | [ ] | ## 13. Timeline | Milestone | Date | |---|---| | *Example:* Plan version 1.0 approved by supervisor and registered | YYYY-MM-DD | | *Example:* Recruitment and baseline complete | YYYY-MM-DD | | *Example:* Week-4 data collection complete | YYYY-MM-DD | | *Example:* Data cleaning complete, blinded data review, data lock | YYYY-MM-DD | | *Example:* Unblinding and primary analysis | YYYY-MM-DD | | *Example:* Analysis report draft (academic/stats-analysis-report) | YYYY-MM-DD | ## Worked example B: a secondary hypothesis (half solved) Write the plan entry for secondary hypothesis 1 (sleep quality, PSQI 0 to 21, higher is worse). 1. Estimand variable: PSQI global score at week [ ]. 2. Intercurrent event "stopped using the app": strategy = [ ], because [ ]. 3. Hypotheses: $H_0: \Delta_{\text{PSQI}} = 0$ vs $H_1$: [ ]. 4. Model: `lm(psqi4 ~ arm + [ ] + year)`, reference arm = [ ]. 5. Multiplicity: tested with [ ] together with WEMWBS, only if [ ]. 6. Missing data: same as the primary, multiple imputation under [ ] with tipping-point sensitivity. ## Exercise C: plan a different study The Lakemoor University library wants to know whether extending opening hours to midnight changes the weekly number of study-room bookings. Six branch libraries are available; three will be chosen at random to extend hours for eight weeks. Bookings are counted weekly for eight weeks before and eight weeks during the change. Write: (1) the estimand with five attributes, (2) the unit of randomisation and why it matters for the analysis, (3) a primary model, (4) one sensitivity analysis. ## Common mistakes and misconceptions - **"Appropriate tests will be used."** Not a plan. Name the model, the covariates, the reference level and the summary. - **Choosing the fallback after seeing results.** "We switched to Mann-Whitney because the t-test was not significant" is a forking path. Decide the rule in section 8. - **Analysing change scores and ANCOVA, then reporting the better one.** Pick one as primary in advance; ANCOVA is usually more precise in randomised designs. - **Testing baseline balance in a randomised trial.** Baseline differences in a randomised study are chance by definition; a p-value for them answers no useful question. Describe baseline, do not test it. - **Subgroup claims from separate p-values.** Use an interaction and its CI. - **Treating per-protocol as the "true" effect.** It conditions on post-randomisation behaviour and can be badly biased. - **Sample size from a published effect.** Published effects are inflated by selection. Use a minimal important difference or a conservative pilot estimate. - **p = 0.03 means a 3% chance that H0 is true.** No: it is the probability of data at least this extreme if H0 and the model were true (Wasserstein and Lazar 2016). ## Self-test questions 1. Name the five attributes of an estimand in the ICH E9(R1) framework. 2. Why should the fallback for a failed assumption be written before the data are seen? 3. A plan needs 64 per group for an unadjusted t-test. With a baseline covariate correlated 0.6 with the outcome, about how many per group does ANCOVA need, before attrition? 4. Three secondary p-values are 0.010, 0.030 and 0.040. Which are significant under Holm at α = 0.05? 5. What is the difference between a sensitivity analysis and a subgroup analysis? ## My notes and next steps - Confidence (1 to 5): estimands [ ], sample size [ ], missing data [ ], multiplicity [ ]. - Open questions for my supervisor: [ ] - Mistakes to log in academic/stats-error-log: [ ] - Next: write the codebook (academic/stats-variable-codebook), then the report shell (academic/stats-analysis-report). ## Sources and further reading Structure aligned with the estimand attributes of ICH E9(R1) and the SAP content items of Gamble et al. (2017), adapted for theses and projects outside clinical trials and written in original words. - ICH (2019). *E9(R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials*. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. - Gamble, C., Krishan, A., Stocken, D., Lewis, S., Juszczak, E., Doré, C., et al. (2017). Guidelines for the content of statistical analysis plans in clinical trials. *JAMA*, 318(23), 2337–2343. - Borm, G. F., Fransen, J., and Lemmens, W. A. J. G. (2007). A simple sample size formula for analysis of covariance in randomized clinical trials. *Journal of Clinical Epidemiology*, 60(12), 1234–1238. - Gelman, A., and Loken, E. (2014). The statistical crisis in science. *American Scientist*, 102(6), 460–465. - Holm, S. (1979). A simple sequentially rejective multiple test procedure. *Scandinavian Journal of Statistics*, 6(2), 65–70. - Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. *Journal of the Royal Statistical Society, Series B*, 57(1), 289–300. - Rubin, D. B. (1987). *Multiple Imputation for Nonresponse in Surveys*. Wiley. - Cohen, J. (1988). *Statistical Power Analysis for the Behavioral Sciences* (2nd ed.). Lawrence Erlbaum Associates. - Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. *The American Statistician*, 70(2), 129–133. - Further reading (by name only): de Smith, M. J. (2024). *Statistical Analysis Handbook* (2024 edition, StatsRef). See also academic/pre-analysis-plan (economics) and science/preregistration (OSF-style form). ## Answer key **Example B.** (1) Week 4. (2) Treatment policy, because the question is the effect of offering the app, whatever use follows. (3) $H_1: \Delta_{\text{PSQI}} \neq 0$ (two-sided). (4) `lm(psqi4 ~ arm + psqi0 + year)`, reference waitlist. (5) Holm, together with WEMWBS, only if the primary hypothesis is significant (gatekeeping). (6) MAR. **Exercise C (one good answer).** (1) Population: the six branch libraries and their weekly bookings; treatment: midnight closing vs usual hours; variable: weekly bookings during the eight change weeks; intercurrent event: a branch closed for repairs, handled by treatment policy (or excluded weeks recorded as a deviation); summary: ratio of mean weekly bookings, extended vs usual, adjusted for the pre-period. (2) The branch is the unit of randomisation, so the effective sample size is six, not the number of weeks; analysing weeks as independent is pseudo-replication. (3) A Poisson or negative binomial model of weekly counts with branch random intercepts (or a branch-level ANCOVA on during-period means with the pre-period mean as covariate), reporting the rate ratio with a 95% CI; see academic/stats-count-regression. (4) A permutation test over the 20 possible assignments of three of six branches. Note its limit: the smallest attainable one-sided p-value is 1/20 = 0.05, so state in the plan that six branches give an estimate with an interval rather than a decisive test. **Self-test.** (1) Population, treatment conditions, variable, handling of intercurrent events, population-level summary. (2) So the choice cannot depend on which option gives the preferred result. (3) $64 \times (1 - 0.36) = 40.96$, about 41 per group. (4) Holm: 0.010 vs 0.05/3 = 0.0167, significant; 0.030 vs 0.05/2 = 0.025, not significant; stop. Only the first. (5) A sensitivity analysis checks whether the main conclusion survives a changed assumption; a subgroup analysis asks whether the effect itself differs between groups. ## Revision history | Version | Date | Author | Change | |---|---|---|---| | 1.0 | YYYY-MM-DD | [Your name] | Plan approved before data lock |