Statistical Analysis Report Template (Statistics, Thesis or Client)

Statistical Analysis Report: [Your study question or client project]

A complete statistical analysis report for a thesis results chapter or a client deliverable. It walks from a plain-language summary through the question, data and methods, descriptive results, the main analysis with tables and a figure, assumption checks, sensitivity analyses, interpretation and limitations, to a reproducibility appendix and a reviewer checklist. The fully filled Lakemoor University example shows what each section should contain and how much to write, and every number in it comes from the R and Python code in the appendix, so you can rerun it and compare. The worked examples use fictional data; replace them with your own. Source basis: APA JARS-Quant reporting standards (by name), the ASA Statement on p-values and standard methods papers, written in original words.

Document control

This table identifies the report, the data version it analyses and the software that produced every number.

Executive summary

Write this for a reader who will read nothing else: a dean, a client, an examiner skimming the chapter. State the question, the answer as a number with its uncertainty, what it means in practice and the most important caveat.

Example: Lakemoor University's Statistics department asked whether a redesigned online homework system raises final exam scores in STAT 101. We compared 18 students in a section that used the new system with 18 students in a section that used the old one, allowing for each student's maths placement score, because the new-system section started with weaker placement scores. After that allowance, students using the new system scored on average 3.6 points higher (out of 100), but the plausible range runs from 2.5 points lower to 9.6 points higher, so the data cannot rule out no benefit. One student who left the exam early through illness strongly influences the result: without that student the estimated benefit is 5.5 points (from 1.4 to 9.6). Because sections were not randomly assigned and the study is small, we recommend a randomised comparison across all sections next term before a decision to adopt the system.

Question and context

Give the research question, why it matters, the decision or theory it informs, and the primary estimand from the analysis plan in one precise sentence.

Example: Does the new homework system improve STAT 101 final exam scores compared with the old system? The department will decide whether to license the system for all sections. Primary estimand: the difference in mean final exam score (new minus old), adjusted for maths placement score, among STAT 101 students in the autumn term.

Data and methods

Describe the design (how units came to be in each group), the sample and exclusions, the measures and how they were scored, and each analysis with its software. Separate primary, secondary and sensitivity analyses, and say which were pre-specified.

Example:

• Design. Observational comparison of two existing sections taught by the same lecturer with the same lectures and exam. Sections were assigned to systems by timetable (not at random): section 1 kept the old system, section 2 used the new one.
• Sample. All 36 enrolled students (18 per section) sat the exam; no exclusions. One student in the new-system section (row 25) left the exam after 40 minutes due to illness, documented in the data cleaning log; per the plan the score stays in the primary analysis.
• Measures. Outcome: final exam score, 0 to 100, marked blind to section. Covariate: maths placement test score, 0 to 100, taken in week 1.
• Primary analysis. Analysis of covariance (ANCOVA): linear regression of exam score on section (reference: old) and placement score, reporting the adjusted difference with a 95% CI. α = 0.05, two-sided.
• Secondary analysis. Unadjusted Welch t-test (Welch 1947) of exam score.
• Assumption checks (pre-specified). Residual normality (Q-Q plot, Shapiro-Wilk), equal residual variance by section (Levene, median-centred), equal slopes (section by placement interaction), influence (Cook's distance > 4/n).
• Sensitivity analyses (pre-specified). S1: exclude points with Cook's distance > 4/n. S2: HC3 robust standard errors. S3: Mann-Whitney U test of exam score (Mann and Whitney 1947).
• Software. R 4.5.1 (stats, car) with a parallel Python 3.14 implementation (scipy, statsmodels); both give the same results (appendix).

The model for the primary analysis is

where $\beta_1$ is the adjusted difference in mean exam score (new minus old) at any fixed placement score.

Descriptive results

Describe the sample before testing anything: group sizes, centre and spread of every analysed variable, and any baseline imbalance in words. In a non-randomised comparison, baseline imbalance is the reason for adjustment.

Example: The new-system section started 8.5 placement points lower than the old-system section. Placement strongly predicts exam score (Pearson r = .72 across all 36 students), so a raw comparison of exam means would mix the effect of the system with the difference in starting ability. The figure shows each student, with a least-squares line per section: at any given placement score, new-system students tend to sit above the old-system line, apart from one low point (placement 55, exam 31).

Main analyses

Report the primary analysis first, exactly as pre-specified, then secondary analyses. For each: the estimate in the units of the outcome, its 95% CI, the test statistic with df and p, and a standardised effect size where it helps.

Example, primary analysis (ANCOVA).

Model fit: $R^2$ = .537, residual SD = 8.22, F(2, 33) = 19.14, p < .001. Adjusted means at the mean placement score (60.08): old system 65.94, 95% CI [61.83, 70.04]; new system 69.51, 95% CI [65.40, 73.61]. Standardised adjusted difference: 3.57 / 11.77 (pooled SD of exam) = 0.30.

Example report sentence: "After adjusting for maths placement, students using the new homework system scored 3.57 points higher on the final exam than students using the old system, 95% CI [-2.46, 9.60], t(33) = 1.20, p = .237, standardised difference 0.30."

Example, secondary analysis (unadjusted). Without adjustment the new-system section scored 3.33 points lower, 95% CI [-11.33, 4.66], Welch t(31.96) = -0.85, p = .402, Cohen's d = -0.28. The sign flips because the new-system section had lower placement scores: this is why the plan made the adjusted analysis primary.

Assumption checks

Report every check the plan named, its result, and the action the plan prescribed. A check that fails is a finding, not an embarrassment: say what it means for the conclusion.

The conclusion from the checks is clear: the model fits the bulk of the data well, and one student's score is far below what placement predicts. That score is genuine (the student was ill), so it is not an error to delete; it is an influential observation whose effect must be shown.

Sensitivity analyses

Put the primary result and every sensitivity analysis in one table, so the reader sees at once whether the conclusion depends on an assumption.

Example: The direction of the adjusted effect is stable (positive in S1 and S2), but its statistical clarity depends on one student: with that student excluded the 95% CI excludes zero. S3 ignores placement and therefore answers the unadjusted question; it agrees with the unadjusted t-test. The primary result stands as the pre-specified answer, and S1 is reported as a sensitivity result, not as a replacement.

Interpretation

Answer the question directly, using the estimate and the interval rather than the p-value alone. A non-significant result is not evidence of no effect when the interval includes practically important values; a significant one is not evidence of an important effect when the interval includes trivial values (see the ASA Statement on p-values, Wasserstein and Lazar 2016).

Example: The best estimate is a benefit of about 3.6 exam points for students of equal placement, and values from a loss of 2.5 points to a gain of 9.6 points are compatible with the data. The department set 5 points as the gain that would justify the licence; the interval includes both zero and values above 5, so this study cannot settle the question either way. The sensitivity result (5.5 points without the ill student) points the same way and suggests the uncertainty is inflated by one unusual score, but it was not the pre-specified analysis.

Limitations

List limitations that could change the conclusion, the most serious first, and for each say how it could bias the estimate.

Example:

1. No randomisation. Sections differ in timetable and possibly in motivation; placement adjustment removes only the measured part of that difference. The estimate could be biased in either direction.
2. Two sections. System and section are fully confounded: any section-level factor (time of day, peer group) is indistinguishable from the system effect, and the 36 students are not 36 independent replications of the system.
3. Small sample. With 18 per group the study has low power for a 5-point effect, so a null result was likely even if the system works.
4. One influential observation. The conclusion's clarity depends on one student; see sensitivity S1.
5. One term, one course. Results may not transfer to other courses or student groups.

Reproducibility appendix

Give the software and versions, the data (or the path and checksum of the locked file), the complete code, and the seed of any random process. This example uses no random numbers; if you add a bootstrap or multiple imputation, set and report the seed. For a full replication package see academic/replication-package-readme.

Example: Software: R 4.5.1 with car 3.1.5; Python 3.14 with numpy 2.5.4, pandas 3.1.0, scipy 1.18.1, statsmodels 0.15.0. Data: typed inline below (36 rows). Seeds: none required. The R output below is abridged: it keeps only the lines the report uses, and some printed tables are condensed onto one labelled line, but every number is copied exactly from the run. The Python block prints its lines in exactly the form shown. Run the code to see the full output.

library(car)   # install.packages("car") once: leveneTest, hccm
d <- data.frame(
  section = factor(rep(c("old", "new"), each = 18), levels = c("old", "new")),
  placement = c(52, 78, 81, 62, 53, 56, 61, 51, 84, 65, 72, 68, 73, 62, 55, 75, 57, 53,
                59, 56, 62, 57, 59, 54, 55, 62, 80, 47, 38, 60, 71, 58, 38, 53, 53, 43),
  exam      = c(65, 73, 74, 76, 63, 59, 63, 50, 90, 70, 80, 79, 72, 74, 64, 80, 53, 64,
                73, 69, 64, 72, 66, 74, 31, 82, 85, 66, 49, 70, 82, 70, 53, 67, 64, 52))
agg <- function(x) c(n = length(x), mean = mean(x), sd = sd(x), median = median(x))
round(t(sapply(split(d$exam, d$section), agg)), 2)
t.test(exam ~ section, data = d)                       # Welch is R's default
fit <- lm(exam ~ section + placement, data = d)        # primary ANCOVA
summary(fit); confint(fit)
shapiro.test(resid(fit))
leveneTest(resid(fit) ~ d$section)
anova(fit, lm(exam ~ section * placement, data = d))   # equal slopes
which.max(cooks.distance(fit)); max(cooks.distance(fit))
rstandard(fit)[25]                                     # standardised residual
s1 <- lm(exam ~ section + placement, data = d[-25, ])  # S1
coef(summary(s1))["sectionnew", ]; confint(s1)["sectionnew", ]
hc3 <- sqrt(diag(hccm(fit, type = "hc3")))              # S2
hc3[2]; coef(fit)[2] / hc3[2]
2 * pt(-abs(coef(fit)[2] / hc3[2]), df = 33)
wilcox.test(exam ~ section, data = d, exact = FALSE)   # S3
predict(fit, data.frame(section = c("old", "new"), placement = mean(d$placement)),
        interval = "confidence")

Abridged R output (numbers exact; layout condensed)
Welch Two Sample t-test: t = 0.84952, df = 31.959, p-value = 0.4019
95 percent confidence interval: -4.659576 11.326243   (old minus new)

            Estimate Std. Error t value Pr(>|t|)
(Intercept)  17.1321     8.8294   1.940   0.0609
sectionnew    3.5711     2.9657   1.204   0.2371
placement     0.8123     0.1339   6.066 7.94e-07
Residual standard error: 8.216 on 33 degrees of freedom
Multiple R-squared:  0.537,  F-statistic: 19.14 on 2 and 33 DF,  p-value: 3.03e-06
sectionnew CI: -2.463  9.605

Shapiro-Wilk: W = 0.8096, p-value = 2.54e-05
Levene (center = median): F = 0.0354, Pr(>F) = 0.8519
Equal slopes: F = 0.1151, Pr(>F) = 0.7366
max Cook's D 0.365 at row 25, standardised residual -4.306
S1 without row 25: estimate 5.525, SE 2.016, t 2.740, p 0.00996, CI 1.418 9.633
S2 HC3 SE 2.868, t 1.245, p 0.222
S3 Wilcoxon rank sum: W = 177.5, p-value = 0.6345
Adjusted means: old 65.94 [61.83, 70.04], new 69.51 [65.40, 73.61]

import numpy as np
import pandas as pd
import scipy.stats as st
import statsmodels.formula.api as smf
from statsmodels.stats.anova import anova_lm

d = pd.DataFrame({
    "section": ["old"] * 18 + ["new"] * 18,
    "placement": [52, 78, 81, 62, 53, 56, 61, 51, 84, 65, 72, 68, 73, 62, 55, 75, 57, 53,
                  59, 56, 62, 57, 59, 54, 55, 62, 80, 47, 38, 60, 71, 58, 38, 53, 53, 43],
    "exam": [65, 73, 74, 76, 63, 59, 63, 50, 90, 70, 80, 79, 72, 74, 64, 80, 53, 64,
             73, 69, 64, 72, 66, 74, 31, 82, 85, 66, 49, 70, 82, 70, 53, 67, 64, 52]})
old, new = d.exam[d.section == "old"], d.exam[d.section == "new"]
w = st.ttest_ind(old, new, equal_var=False)           # Welch must be asked for
pooled = st.ttest_ind(old, new)                        # scipy default: pooled
print(f"Welch: statistic={w.statistic:.4f}, pvalue={w.pvalue:.4f}, df={w.df:.4f}")
print(f"scipy default (pooled): statistic={pooled.statistic:.4f}, pvalue={pooled.pvalue:.4f}, df={pooled.df}")
f = "exam ~ C(section, Treatment('old')) + placement"
fit = smf.ols(f, data=d).fit()
print(fit.summary2().tables[1].round(4).to_string())
sw = st.shapiro(fit.resid)
print(f"Shapiro-Wilk: W = {sw.statistic:.4f}, p = {sw.pvalue:.3g}")
lev = st.levene(fit.resid[d.section == "old"], fit.resid[d.section == "new"])  # center="median"
print(f"Levene (median): F = {lev.statistic:.4f}, p = {lev.pvalue:.4f}")
slopes = anova_lm(fit, smf.ols("exam ~ C(section, Treatment('old')) * placement", data=d).fit())
print(f"Equal slopes: F = {slopes.F[1]:.6f}, Pr(>F) = {slopes['Pr(>F)'][1]:.6f}")
print("max Cook's D", fit.get_influence().cooks_distance[0].max().round(3))
s1 = smf.ols(f, data=d.drop(index=24)).fit()            # row 25 is index 24
print("S1:", s1.params.iloc[1].round(3), s1.conf_int().iloc[1].round(3).tolist())
print("S2 HC3 z p:", smf.ols(f, data=d).fit(cov_type="HC3").pvalues.iloc[1].round(3))
print("S2 HC3 t p:", smf.ols(f, data=d).fit(cov_type="HC3", use_t=True).pvalues.iloc[1].round(3))
mw = st.mannwhitneyu(old, new)
print(f"Mann-Whitney: U = {mw.statistic}, p = {mw.pvalue:.4f}")

Welch: statistic=0.8495, pvalue=0.4019, df=31.9587
scipy default (pooled): statistic=0.8495, pvalue=0.4015, df=34.0
                                       Coef.  Std.Err.       t   P>|t|  [0.025   0.975]
Intercept                            17.1321    8.8294  1.9404  0.0609 -0.8313  35.0956
C(section, Treatment('old'))[T.new]   3.5711    2.9657  1.2041  0.2371 -2.4626   9.6047
placement                             0.8123    0.1339  6.0662  0.0000  0.5399   1.0847
Shapiro-Wilk: W = 0.8096, p = 2.54e-05
Levene (median): F = 0.0354, p = 0.8519
Equal slopes: F = 0.115137, Pr(>F) = 0.736587
max Cook's D 0.365
S1: 5.525 [1.418, 9.633]
S2 HC3 z p: 0.213
S2 HC3 t p: 0.222
Mann-Whitney: U = 177.5, p = 0.6345

Where R and Python differ by design. (1) R's t.test uses Welch by default; scipy's ttest_ind uses the pooled test unless you pass equal_var=False (here p = .4019 vs .4015, but they can differ a lot with unequal variances). (2) statsmodels reports HC3 p-values from the normal distribution unless you pass use_t=True (.213 vs .222); the report uses the t version to match R. (3) Both tools pick the reference level alphabetically by default ("new" before "old"); the code fixes "old" as reference in both, otherwise the sign of the estimate flips.

Reviewer checklist

Tick each item before the report leaves your desk, then ask a reviewer to tick it independently.

• The executive summary states the main estimate with its uncertainty in plain language, and the main caveat.
• The primary analysis is the one in the approved plan, reported first; any deviation is logged and explained.
• Every number in the text appears in the appendix output (spot-check five).
• Descriptives are given by group for every analysed variable, with n.
• Each estimate has units, a 95% CI, the test statistic with df, and an exact p (three decimals or < .001).
• An effect size is reported and interpreted in context, not only by a generic label.
• All pre-specified assumption checks are reported with the action taken.
• Sensitivity analyses appear next to the primary result, and the text says whether the conclusion changes.
• Non-significant results are not described as "no effect"; significant results are not described as "proof".
• Causal language matches the design (no causal claim from a non-randomised comparison without caveats).
• Limitations say in which direction each could bias the estimate.
• Software, versions, data version and seeds are listed; the code runs from a clean session.
• Figures have axis titles with units and are readable in greyscale.

Worked example B: write up sensitivity analysis S1 (half solved)

Complete the paragraph that reports sensitivity analysis S1.

"Excluding the one student with Cook's distance above 4/n (row [ ], distance [ ]), the adjusted difference was [ ] points, 95% CI [[ ], 9.63], t(32) = 2.74, p = [ ]. The direction matches the primary analysis, but unlike the primary analysis the interval [ ] zero. Because this exclusion was [pre-specified as a sensitivity analysis / chosen after seeing the data], it [does / does not] replace the primary result."

Explain in one sentence why the df is 32 here and 33 in the primary analysis: [ ].

Exercise C: write the executive summary for a different result

Harbour City Library ran a randomised trial of a text-message reminder on returning books on time. Primary result: on-time return 78% with reminders vs 71% without, risk difference 7.0 percentage points, 95% CI [1.3, 12.7], p = .016, n = 900 (450 per arm). A pre-specified sensitivity analysis that treats lost cards as late returns gives 6.1 points, 95% CI [0.4, 11.8]. Write a five-sentence executive summary: question, design, answer with uncertainty, meaning, caveat.

Common mistakes and misconceptions

• Leading with the p-value. Lead with the effect and its interval; the p-value is supporting detail.
• "Not significant, so no effect". Example A's interval includes a 9.6-point gain. Absence of evidence is not evidence of absence.
• Swapping the primary analysis after seeing results. Promoting S1 to primary because it is significant would be a forking path. Report it as sensitivity.
• Deleting an awkward data point. A genuine but unusual score stays in the primary analysis; show its influence.
• Testing baseline differences and calling the groups "comparable". Report the difference and its consequence; a non-significant baseline test does not prove balance.
• Numbers that do not match the output. Copy from the output, or generate tables by code.
• Causal verbs for an observational comparison. "The system raised scores" overstates a non-randomised design; "students using the system scored higher, after adjustment" is accurate.
• p = 0.03 means a 3% chance that H0 is true. It is the probability of data at least this extreme if H0 and the model assumptions held.

Self-test questions

1. In the example, why does the unadjusted difference have the opposite sign to the adjusted difference?
2. Which result belongs in the abstract: the primary ANCOVA or sensitivity S1? Why?
3. The Shapiro-Wilk test rejected normality. Name two appropriate responses that were pre-specified.
4. Why do scipy's ttest_ind and R's t.test give different p-values by default?
5. Rewrite this sentence correctly: "The new system had no effect (p = .237)."

My notes and next steps

• Confidence (1 to 5): writing estimates with CIs [ ], assumption reporting [ ], sensitivity tables [ ], limitations [ ].
• Questions for my supervisor or client: [ ]
• Mistakes to log in academic/stats-error-log: [ ]
• Next: academic/stats-results-reporting for sentence templates; academic/replication-package-readme for the code.

Sources and further reading

Structure aligned with the APA Journal Article Reporting Standards for quantitative research (JARS-Quant, by name) and the ASA Statement on Statistical Significance and P-Values, with methods cited from their original papers; all text and examples are original.

• Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., and Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 3–25.
• American Psychological Association (2020). Publication Manual of the American Psychological Association (7th ed.), by name only.
• Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133.
• Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1–2), 28–35.
• Mann, H. B., and Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18(1), 50–60.
• Shapiro, S. S., and Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3–4), 591–611.
• Cook, R. D. (1977). Detection of influential observation in linear regression. Technometrics, 19(1), 15–18.
• Long, J. S., and Ervin, L. H. (2000). Using heteroscedasticity consistent standard errors in the linear regression model. The American Statistician, 54(3), 217–224.
• Further reading (by name only): de Smith, M. J. (2024). Statistical Analysis Handbook (2024 edition, StatsRef).

Answer key

Example B. Row 25, distance 0.365; 5.53 points, 95% CI [1.42, 9.63], p = .010; the interval excludes zero; pre-specified as a sensitivity analysis, so it does not replace the primary result. The df is 32 because one observation was removed (35 students minus 3 estimated coefficients), versus 36 minus 3 = 33.

Exercise C (model answer). "Harbour City Library tested whether a text-message reminder helps members return books on time. We randomly assigned 900 members to receive reminders or not. With reminders, 78% returned books on time compared with 71% without, a gain of 7 percentage points; the true gain is plausibly between about 1 and 13 points. Even counting lost cards as late returns, the gain stays positive (about 6 points). The size of the benefit is uncertain, so the library should weigh the messaging cost against a gain that could be as small as 1 point."

Self-test. (1) The new-system section had lower placement scores, and placement strongly predicts exam score; adjustment compares students of equal placement. (2) The primary ANCOVA, because it was pre-specified; S1 is reported alongside as a sensitivity result. (3) Showing the influence of the outlying point (S1), and a rank-based test that does not rely on normality (S3, which answers the unadjusted question). S2 (HC3 robust SE) protects against unequal variance, not non-normality, so it is not a response to this check. With n = 36 and a non-normality driven by one point, the t-based inference for the coefficient is fairly robust; the influence analysis matters more. (4) scipy defaults to the pooled-variance test, R to Welch. (5) "After adjustment, the new system was associated with a 3.6-point higher score, 95% CI [-2.5, 9.6], p = .237; the data are compatible with no benefit and with a benefit above 5 points."

Revision history