# ANCOVA: [Your study question or course] > A method card for analysis of covariance (ANCOVA): comparing group means on an outcome while adjusting for a > continuous covariate measured before treatment, such as a pre-test score. It covers adjusted means (by hand and > with emmeans), the assumptions including homogeneity of regression slopes, ANCOVA written as a linear model in R > (`lm`) and Python (statsmodels `ols`), pre-post designs (ANCOVA versus change scores), Lord's paradox, and > reporting. It is for students preparing for a course or exam, thesis writers with baseline and follow-up data, > and analysts evaluating interventions. It speeds up learning with one fixed order of decisions, a full worked > example, a half-solved example, an exercise where the key assumption fails, and a self-test with a separate > answer key. The worked examples use fictional data; replace them with your own. > Source basis: standard linear-model methods (Fisher 1932; Lord 1967; Searle, Speed and Milliken 1980; Senn 2006) > and the NIST/SEMATECH e-Handbook, written in original words. ## Document control | Field | Value | |---|---| | Course or project | [Course code and name, or thesis chapter] | | Learner or analyst | [Your name] | | Data set | [Name, source, n per group, covariate and when it was measured] | | Software and versions | R 4.5.1 (stats), car 3.1.5, emmeans 2.0.4; Python 3.14, pandas 3.1.0, statsmodels 0.15.0 | | Date | YYYY-MM-DD | | Version | [0.1] | ## When to use it **One-sentence rule:** use ANCOVA when you compare the means of two or more groups on a continuous outcome and have a continuous covariate, measured before treatment, that predicts the outcome; it removes covariate-related noise (more power) and corrects for chance baseline imbalance (fairer comparison). ANCOVA is simply a linear regression with one categorical predictor (the group) and one continuous predictor (the covariate). Once you see it that way, everything else follows: the group coefficient is the adjusted difference, the covariate coefficient is the common slope, and the assumptions are regression assumptions plus one extra, that the slope is the same in every group. | Use when | Do not use when | |---|---| | Groups were randomised and you have a baseline measure of the outcome (pre-test) | The covariate was measured after treatment and could be affected by it | | The covariate is related to the outcome (correlation above about 0.3) | The covariate is unrelated to the outcome (no gain, lost df) | | The relationship between covariate and outcome is roughly linear | The relationship is curved (add a quadratic term or transform) | | Regression slopes are similar across groups (interaction not significant and small) | Slopes clearly differ (report the interaction and simple slopes instead) | | You want to "equate" randomised groups for chance baseline differences | Groups are intact, non-randomised and differ systematically on the covariate (adjustment cannot make them comparable) | **Alternatives:** one-way ANOVA on the outcome (no covariate, less power); Welch t-test on change scores (simpler, less efficient unless the slope is near 1); regression with an interaction (when slopes differ, Johnson-Neyman intervals); linear mixed model (several follow-up time points, see stats-mixed-models); propensity-score methods (observational group comparisons). ## The question and the data **Research question (Example A):** after adjusting for the pre-test, do STAT 101 students who received peer tutoring score higher on the post-test than students with standard support? Twenty students at Lakemoor University (fictional) were randomly assigned to standard support or weekly peer tutoring (10 each). All took a 0 to 100 pre-test in week 1, before the assignment, and a post-test in week 10. | Variable | Role | Measurement level | Values | |---|---|---|---| | group | Factor (treatment) | Nominal, 2 levels | standard, tutoring | | pre | Covariate (measured before assignment) | Interval | 0 to 100 | | post | Outcome | Interval | 0 to 100 | | Student | group | pre | post | |---|---|---|---| | *Example:* 1 | standard | 52 | 56 | | 2 | standard | 61 | 67 | | 3 | standard | 48 | 50 | | 4 | standard | 70 | 72 | | 5 | standard | 57 | 62 | | 6 | standard | 66 | 70 | | 7 | standard | 45 | 49 | | 8 | standard | 59 | 64 | | 9 | standard | 63 | 66 | | 10 | standard | 55 | 60 | | 11 | tutoring | 50 | 60 | | 12 | tutoring | 58 | 64 | | 13 | tutoring | 47 | 56 | | 14 | tutoring | 64 | 70 | | 15 | tutoring | 53 | 61 | | 16 | tutoring | 68 | 77 | | 17 | tutoring | 44 | 53 | | 18 | tutoring | 56 | 63 | | 19 | tutoring | 60 | 71 | | 20 | tutoring | 49 | 58 | | Group | Mean pre | Mean post | SD post | |---|---|---|---| | standard | 57.6 | 61.6 | 7.89 | | tutoring | 54.9 | 63.3 | 7.42 | ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "description": "Post-test against pre-test for standard and tutoring groups with a fitted line per group", "width": 360, "height": 260, "data": {"values": [ {"group": "standard", "pre": 52, "post": 56}, {"group": "standard", "pre": 61, "post": 67}, {"group": "standard", "pre": 48, "post": 50}, {"group": "standard", "pre": 70, "post": 72}, {"group": "standard", "pre": 57, "post": 62}, {"group": "standard", "pre": 66, "post": 70}, {"group": "standard", "pre": 45, "post": 49}, {"group": "standard", "pre": 59, "post": 64}, {"group": "standard", "pre": 63, "post": 66}, {"group": "standard", "pre": 55, "post": 60}, {"group": "tutoring", "pre": 50, "post": 60}, {"group": "tutoring", "pre": 58, "post": 64}, {"group": "tutoring", "pre": 47, "post": 56}, {"group": "tutoring", "pre": 64, "post": 70}, {"group": "tutoring", "pre": 53, "post": 61}, {"group": "tutoring", "pre": 68, "post": 77}, {"group": "tutoring", "pre": 44, "post": 53}, {"group": "tutoring", "pre": 56, "post": 63}, {"group": "tutoring", "pre": 60, "post": 71}, {"group": "tutoring", "pre": 49, "post": 58} ]}, "layer": [ { "mark": {"type": "point", "filled": true, "size": 60}, "encoding": { "x": {"field": "pre", "type": "quantitative", "title": "Pre-test score", "scale": {"domain": [40, 75]}}, "y": {"field": "post", "type": "quantitative", "title": "Post-test score", "scale": {"domain": [45, 80]}}, "color": {"field": "group", "type": "nominal", "title": "Group"} } }, { "transform": [{"regression": "post", "on": "pre", "groupby": ["group"]}], "mark": "line", "encoding": { "x": {"field": "pre", "type": "quantitative"}, "y": {"field": "post", "type": "quantitative"}, "color": {"field": "group", "type": "nominal"} } } ] } ``` Read the chart first. The two lines are nearly parallel (homogeneity of slopes looks plausible), and the tutoring line sits about 4 points higher at every pre-test value. Yet the raw post-test means differ by only 1.7 points, because the tutoring group happened to start 2.7 points lower. ANCOVA turns that picture into a number. ## Assumptions and how to check each | Assumption | How to check | R | Python | If violated | |---|---|---|---|---| | Covariate measured before treatment, not affected by it | Design and timeline | Study design | Study design | Do not adjust for it; report ANOVA | | Independence of observations | Design: one row per unit, no clustering | Study design | Study design | Mixed model with a cluster random effect | | Linear covariate-outcome relationship in each group | Scatter plot with group lines; residuals vs covariate | `plot(d$pre, resid(fit))` | `plt.scatter(d.pre, fit.resid)` | Add `I(pre^2)` or transform | | Homogeneity of regression slopes | Test the covariate by group interaction | `anova(lm(post ~ pre * group))` | `anova_lm(ols("post ~ pre * group"), typ=2)` | Keep the interaction; compare groups at chosen covariate values | | Normal residuals | Q-Q plot, Shapiro-Wilk on residuals | `shapiro.test(resid(fit))` | `stats.shapiro(fit.resid)` | Robust SEs, transformation, or a rank-based approach | | Equal residual variance across groups | Residual plot by group; Levene | `car::leveneTest(post ~ group)` | `stats.levene(...)` | HC3 robust standard errors | | Covariate measured reliably | Reliability of the pre-test | Prior evidence | Prior evidence | Measurement error biases the adjustment; say so | | Groups do not differ systematically on the covariate (non-randomised data) | Compare covariate means | `t.test(pre ~ group)` | `stats.ttest_ind(..., equal_var=False)` | Interpret adjusted means cautiously (Lord's paradox) | **A note on testing baseline balance.** In a randomised study any baseline difference is due to chance by definition, so a significance test of the pre-test is not needed to justify ANCOVA; adjust for the pre-specified covariate regardless. In Example A the check is shown only so you can see that the 2.7-point gap is well within chance (Welch *t*(18.0) = 0.78, *p* = .448). ## How it works **Plain language.** Imagine sliding every student along their group's regression line until they all have the same pre-test score, the overall average. The group means after that slide are the **adjusted means**: the means you would expect if both groups had started at the same level. A group that started lower is moved up a little; a group that started higher is moved down a little. How far each group moves depends on the slope (how much the outcome changes per point of covariate) and on how far the group's covariate mean is from the grand mean. The test of the group effect then compares adjusted means against the residual variation around the lines, which is much smaller than the raw variation in the outcome when the covariate is a good predictor. That is where the gain in power comes from. The ANCOVA model with groups $j = 1, \dots, g$ and units $i$: $$ y_{ij} = \mu + \tau_j + \beta\,(x_{ij} - \bar{x}) + \varepsilon_{ij}, \qquad \varepsilon_{ij} \sim N(0, \sigma^2) $$ The common slope is estimated from the pooled within-group sums of squares and cross-products: $$ b_w = \frac{SP_{xy,\text{within}}}{SS_{x,\text{within}}} = \frac{\sum_j \sum_i (x_{ij} - \bar{x}_j)(y_{ij} - \bar{y}_j)}{\sum_j \sum_i (x_{ij} - \bar{x}_j)^2} $$ and the adjusted mean of group $j$ is $$ \bar{y}_{j,\text{adj}} = \bar{y}_j - b_w\,(\bar{x}_j - \bar{x}) $$ The F test for groups compares a model with group and covariate against a model with the covariate only; with two groups it equals the squared t statistic of the group coefficient. Partial eta squared is $SS_{\text{group}} / (SS_{\text{group}} + SS_{\text{residual}})$. **ANCOVA versus change scores.** Analysing the change $y - x$ is the special case of ANCOVA with the slope fixed at 1. ANCOVA estimates the slope from the data instead, so it is at least as precise when groups are randomised, and more precise when the true slope is far from 1 (Senn 2006; Vickers and Altman 2001). Analysing only the post-test is the special case with the slope fixed at 0. | Symbol | Meaning | Example A value | |---|---|---| | $y_{ij}$ | Outcome for unit $i$ in group $j$ | post-test | | $x_{ij}$ | Covariate | pre-test | | $\bar{x}$ | Grand mean of the covariate | 56.25 | | $\bar{x}_j$ | Covariate mean in group $j$ | 57.6, 54.9 | | $b_w$ | Pooled within-group slope | 1053.7 / 1091.3 = 0.966 | | $\bar{y}_{j,\text{adj}}$ | Adjusted mean | 60.30, 64.60 | | $\tau_j$ | Group effect (adjusted difference) | 4.31 | | $\eta^2_p$ | Partial eta squared for group | 0.70 | ```mermaid flowchart TD A["Covariate measured before treatment?"] -->|No| B["Do not adjust: ANOVA or redesign"] A -->|Yes| C["Covariate related to outcome?"] C -->|No| D["One-way ANOVA or Welch t-test"] C -->|Yes| E["Plot outcome vs covariate by group"] E --> F["Fit model with covariate x group interaction"] F --> G["Interaction significant or slopes clearly differ?"] G -->|Yes| H["Report slopes per group, compare at chosen covariate values"] G -->|No| I["Fit ANCOVA: outcome ~ covariate + group"] I --> J["Check residuals: normality, equal variance"] J --> K["Adjusted means with CI, pairwise contrasts with Holm or Tukey"] K --> L["Effect size and APA report"] ``` ## Worked example A (full) **Hypotheses.** $H_0$: the adjusted post-test means are equal ($\tau_{\text{tutoring}} = \tau_{\text{standard}}$). $H_1$: they differ. α = 0.05, two-sided. **R code.** ```r # Lakemoor University: STAT 101 pre-test and post-test (0-100), standard vs peer tutoring d <- data.frame( group = factor(rep(c("standard", "tutoring"), each = 10)), pre = c(52, 61, 48, 70, 57, 66, 45, 59, 63, 55, 50, 58, 47, 64, 53, 68, 44, 56, 60, 49), post = c(56, 67, 50, 72, 62, 70, 49, 64, 66, 60, 60, 64, 56, 70, 61, 77, 53, 63, 71, 58) ) aggregate(cbind(pre, post) ~ group, data = d, FUN = mean) # 1. Homogeneity of regression slopes: test the pre x group interaction anova(lm(post ~ pre * group, data = d)) # 2. ANCOVA as a linear model (covariate first, then group) fit <- lm(post ~ pre + group, data = d) summary(fit) confint(fit) anova(fit) # sequential (type I) SS # 3. Adjusted means by hand: mean_g - b * (mean_pre_g - grand mean_pre) b <- coef(fit)["pre"] m <- aggregate(cbind(pre, post) ~ group, data = d, FUN = mean) m$adj <- m$post - b * (m$pre - mean(d$pre)) m # 4. Adjusted means with emmeans (install.packages("emmeans") once) library(emmeans) emm <- emmeans(fit, ~ group) emm pairs(emm) # 5. Effect size: partial eta squared for group a <- anova(fit) a["group", "Sum Sq"] / (a["group", "Sum Sq"] + a["Residuals", "Sum Sq"]) # 6. Comparison: change scores and post-test only t.test(I(post - pre) ~ group, data = d) # Welch on change scores t.test(post ~ group, data = d) # Welch on post-test only ``` ```text (output abridged) group pre post 1 standard 57.6 61.6 2 tutoring 54.9 63.3 Response: post # slopes check Df Sum Sq Mean Sq F value Pr(>F) pre 1 942.09 942.09 391.8400 1.122e-12 *** group 1 89.75 89.75 37.3302 1.506e-05 *** pre:group 1 0.64 0.64 0.2645 0.614 Residuals 16 38.47 2.40 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 5.98457 2.68761 2.227 0.0398 * pre 0.96555 0.04591 21.031 1.31e-13 *** grouptutoring 4.30697 0.68951 6.246 8.86e-06 *** Residual standard error: 1.517 on 17 degrees of freedom Multiple R-squared: 0.9635, Adjusted R-squared: 0.9592 2.5 % 97.5 % (Intercept) 0.3141975 11.654940 pre 0.8686819 1.062409 grouptutoring 2.8522419 5.761705 Response: post # ANCOVA, type I SS Df Sum Sq Mean Sq F value Pr(>F) pre 1 942.09 942.09 409.558 2.467e-13 *** group 1 89.75 89.75 39.018 8.859e-06 *** Residuals 17 39.10 2.30 group pre post adj 1 standard 57.6 61.6 60.29651 2 tutoring 54.9 63.3 64.60349 group emmean SE df lower.CL upper.CL standard 60.3 0.484 17 59.3 61.3 tutoring 64.6 0.484 17 63.6 65.6 contrast estimate SE df t.ratio p.value standard - tutoring -4.31 0.69 17 -6.246 <0.0001 [1] 0.6965274 # partial eta squared Welch Two Sample t-test data: I(post - pre) by group t = -6.5672, df = 17.254, p-value = 4.45e-06 95 percent confidence interval: -5.811975 -2.988025 mean in group standard mean in group tutoring 4.0 8.4 Welch Two Sample t-test data: post by group t = -0.49618, df = 17.934, p-value = 0.6258 95 percent confidence interval: -8.900104 5.500104 mean in group standard mean in group tutoring 61.6 63.3 ``` **Python code.** ```python import pandas as pd import statsmodels.formula.api as smf from statsmodels.stats.anova import anova_lm d = pd.DataFrame({ "group": ["standard"] * 10 + ["tutoring"] * 10, "pre": [52, 61, 48, 70, 57, 66, 45, 59, 63, 55, 50, 58, 47, 64, 53, 68, 44, 56, 60, 49], "post": [56, 67, 50, 72, 62, 70, 49, 64, 66, 60, 60, 64, 56, 70, 61, 77, 53, 63, 71, 58], }) # 1. Homogeneity of slopes print(anova_lm(smf.ols("post ~ pre * group", data=d).fit(), typ=2).round(4)) # 2. ANCOVA (type II SS: each term adjusted for the other) fit = smf.ols("post ~ pre + C(group)", data=d).fit() print(fit.summary().tables[1]) aov = anova_lm(fit, typ=2) print(aov.round(4)) ss = aov["sum_sq"] print("partial eta^2 (group):", round(ss["C(group)"] / (ss["C(group)"] + ss["Residual"]), 4)) # 3. Adjusted means: predict each group at the grand mean of the covariate grid = pd.DataFrame({"group": ["standard", "tutoring"], "pre": d["pre"].mean()}) pred = fit.get_prediction(grid).summary_frame(alpha=0.05) print(pd.concat([grid, pred[["mean", "mean_se", "mean_ci_lower", "mean_ci_upper"]]], axis=1).round(3)) ``` ```text sum_sq df F PR(>F) group 89.7523 1.0 37.3302 0.000 pre 1017.3955 1.0 423.1601 0.000 pre:group 0.6360 1.0 0.2645 0.614 Residual 38.4685 16.0 NaN NaN ======================================================================================== coef std err t P>|t| [0.025 0.975] ---------------------------------------------------------------------------------------- Intercept 5.9846 2.688 2.227 0.040 0.314 11.655 C(group)[T.tutoring] 4.3070 0.690 6.246 0.000 2.852 5.762 pre 0.9655 0.046 21.031 0.000 0.869 1.062 ======================================================================================== sum_sq df F PR(>F) C(group) 89.7523 1.0 39.0182 0.0 pre 1017.3955 1.0 442.2948 0.0 Residual 39.1045 17.0 NaN NaN partial eta^2 (group): 0.6965 group pre mean mean_se mean_ci_lower mean_ci_upper 0 standard 56.25 60.297 0.484 59.276 61.317 1 tutoring 56.25 64.603 0.484 63.583 65.624 ``` **Where R and Python differ, and why.** The coefficients, the group test and the adjusted means are identical. The sum of squares for the covariate is not: R's `anova()` uses sequential (type I) sums of squares, so `pre`, entered first, gets 942.09 without adjusting for group. Python's `anova_lm(typ=2)` adjusts each term for the other, so `pre` gets 1017.40. The group row is the same in both (89.75) because group was entered last in R. If you put group first in the R formula, its type I SS changes and no longer tests the adjusted difference. Rule: in ANCOVA, enter the covariate first, or use type II (`car::Anova(fit, type = 2)` gives 1017.40 and 89.75, as Python does). **Calculation by hand for the key statistic.** - Pooled within-group slope: $b_w = 1053.7 / 1091.3 = 0.9655$ (the same as the `pre` coefficient). - Grand mean of the pre-test: $\bar{x} = 56.25$. - Standard: $61.6 - 0.9655 \times (57.6 - 56.25) = 61.6 - 1.30 = 60.30$. - Tutoring: $63.3 - 0.9655 \times (54.9 - 56.25) = 63.3 + 1.30 = 64.60$. - Adjusted difference: $64.60 - 60.30 = 4.31$, the `grouptutoring` coefficient. - Test: $t = 4.307 / 0.690 = 6.25$ on 17 df, and $t^2 = 39.0 = F$. **Effect size with CI.** The adjusted difference is 4.31 points, 95% CI [2.85, 5.76]. Partial η² for group is 89.75 / (89.75 + 39.10) = 0.70. Beware: partial η² is inflated by the covariate, because the covariate removes most of the residual variation. A standardised difference that is comparable with studies without a covariate divides the adjusted difference by the pooled raw SD of the outcome: $4.31 / 7.66 = 0.56$. Report both and say which denominator you used. **Three analyses, three answers.** The post-test-only comparison finds nothing (difference 1.7, *p* = .626), because the raw post-test SD of about 7.7 swamps the effect and the tutoring group started lower. The change-score analysis gives a difference of 4.4 (95% CI [2.99, 5.81]). ANCOVA gives 4.31 (95% CI [2.85, 5.76]). Because the estimated slope (0.97) is close to 1, change scores and ANCOVA agree closely here; with a slope far from 1 they would not, and ANCOVA would be the more precise of the two. **Interpretation.** Adjusting for the pre-test, students who received peer tutoring scored about 4.3 points higher on the post-test than students with standard support. The covariate explained most of the variation in post-test scores, which is why the adjusted comparison is so much more precise than the raw one. Because assignment was random, the adjusted difference estimates the causal effect of tutoring in this sample. **APA-style report.** An ANCOVA with pre-test score as covariate showed that post-test scores differed between groups, *F*(1, 17) = 39.02, *p* < .001, partial η² = .70. The covariate-by-group interaction was not significant, *F*(1, 16) = 0.26, *p* = .614, supporting homogeneity of slopes. Adjusted for pre-test (*M* = 56.25), the tutoring group scored higher (adjusted *M* = 64.60, 95% CI [63.58, 65.62]) than the standard group (adjusted *M* = 60.30, 95% CI [59.28, 61.32]), adjusted difference = 4.31, 95% CI [2.85, 5.76]. ## Worked example B (half solved) **Setting.** Brightwell Bakery (fictional) tested three oven settings (A, B, C) on 15 sourdough loaves, five per setting. Dough weight (g) was recorded before baking, because heavier dough gives larger loaves. Outcome: loaf volume (mL). | Oven | Dough weight (g) | Volume (mL) | |---|---|---| | A | 980, 1010, 995, 1020, 970 | 2420, 2465, 2450, 2510, 2380 | | B | 1000, 990, 1030, 985, 1015 | 2505, 2500, 2570, 2495, 2560 | | C | 1005, 975, 1025, 990, 1010 | 2490, 2390, 2515, 2460, 2480 | ```r library(car); library(emmeans) b <- data.frame( oven = factor(rep(c("A", "B", "C"), each = 5)), dough = c(980, 1010, 995, 1020, 970, 1000, 990, 1030, 985, 1015, 1005, 975, 1025, 990, 1010), vol = c(2420, 2465, 2450, 2510, 2380, 2505, 2500, 2570, 2495, 2560, 2490, 2390, 2515, 2460, 2480) ) aggregate(cbind(dough, vol) ~ oven, data = b, FUN = mean) anova(lm(vol ~ dough * oven, data = b)) # slopes check fb <- lm(vol ~ dough + oven, data = b) Anova(fb, type = 2) coef(fb) emm <- emmeans(fb, ~ oven) emm pairs(emm, adjust = "holm") ``` ```text (output abridged) oven dough vol 1 A 995 2445 2 B 1004 2526 3 C 1001 2467 Df Sum Sq Mean Sq F value Pr(>F) dough 1 28573.2 28573.2 145.0563 7.464e-07 *** oven 2 10645.6 5322.8 27.0220 0.0001569 *** dough:oven 2 201.8 100.9 0.5121 0.6156739 Residuals 9 1772.8 197.0 Anova Table (Type II tests) Sum Sq Df F value Pr(>F) dough 21675.4 1 120.749 2.860e-07 *** oven 10645.6 2 29.652 3.709e-05 *** Residuals 1974.6 11 (Intercept) dough ovenB ovenC 270.903084 2.185022 61.334802 8.889868 oven emmean SE df lower.CL upper.CL A 2456 6.07 11 2443 2469 B 2517 6.04 11 2504 2531 C 2465 6.00 11 2452 2478 contrast estimate SE df t.ratio p.value A - B -61.33 8.66 11 -7.082 <0.0001 A - C -8.89 8.56 11 -1.039 0.3212 B - C 52.44 8.49 11 6.174 0.0001 P value adjustment: holm method for 3 tests ``` The Python equivalent is `anova_lm(smf.ols("vol ~ dough + C(oven)", data=b).fit(), typ=2)`, which reproduces the type II table above. 1. Slopes check: the dough-by-oven interaction gives *F*(2, 9) = [ ], *p* = [ ]. Conclusion: [ ]. 2. The grand mean of dough weight is 1000 g, and the common slope is 2.185 mL per g. Adjusted mean for oven A by hand: $2445 - 2.185 \times (995 - 1000) =$ [ ]. For oven B: $2526 - 2.185 \times ($ [ ] $- 1000) =$ [ ]. 3. Why does oven A's mean go up after adjustment while oven B's goes down? [ ] 4. Partial η² for oven = 10645.6 / (10645.6 + [ ]) = [ ]. 5. Which pairs differ after Holm adjustment? [ ] 6. Report: An ANCOVA with dough weight as covariate showed that loaf volume differed between oven settings, *F*([ ], [ ]) = [ ], *p* < .001, partial η² = [ ]. [Adjusted means and pairwise result.] ## Exercise C (unsolved) Sixteen Lakemoor University students took a statistics lab in one of two formats, guided or open-ended (eight each). A maths placement score (0 to 100) was recorded at enrolment; the outcome is the lab score (0 to 100). | Format | Maths placement | Lab score | |---|---|---| | guided | 40, 45, 50, 55, 60, 65, 70, 75 | 61, 64, 65, 67, 66, 70, 69, 73 | | open | 42, 47, 52, 57, 62, 67, 72, 77 | 52, 54, 61, 64, 71, 74, 80, 86 | Tasks: (a) plot lab score against maths by format; (b) test homogeneity of slopes; (c) decide whether a standard ANCOVA is appropriate; (d) if not, estimate the slope in each format and compare the formats at maths = 45 and at maths = 70 (hint: `emtrends()` and `emmeans(..., at = list(maths = c(45, 70)))`); (e) find the maths score at which the two lines cross; (f) write two sentences for a results section. ## Reading software output | Output line | What it means | What to report | |---|---|---| | `pre:group` row in the interaction model | Test of homogeneity of slopes | F, df and p, as an assumption check | | `pre` coefficient | Common within-group slope of outcome on covariate | Optional; shows the covariate mattered | | `grouptutoring` coefficient | Adjusted difference (tutoring minus reference) at any covariate value | Estimate with 95% CI | | `(Intercept)` | Predicted outcome for the reference group at covariate = 0 | Rarely meaningful; centre the covariate if you want it to be | | Type I `anova()` table | Sequential SS; depends on term order | Only if the covariate was entered first | | Type II table (`car::Anova`, `anova_lm(typ=2)`) | Each term adjusted for the other | F for group, df, p, partial η² | | `emmeans` table | Adjusted means at the covariate mean, with SE and CI | Adjusted means and CIs | | `pairs()` table | Pairwise adjusted differences with the chosen adjustment | Estimates, adjusted p | | Multiple R-squared | Variance explained by covariate and group together | Optional | ## Reporting template **Fill-in sentence.** An ANCOVA with [covariate, measured at baseline] as covariate showed that [outcome] [did / did not] differ between [groups], *F*([df1], [df2]) = [F], *p* = [p], partial η² = [value]. The covariate-by-group interaction was not significant, *F*([df], [df]) = [F], *p* = [p]. Adjusted for [covariate] (*M* = [value]), [group 1] (adjusted *M* = [value], 95% CI [[lower], [upper]]) [scored higher / lower than] [group 2] (adjusted *M* = [value], 95% CI [[lower], [upper]]), adjusted difference = [value], 95% CI [[lower], [upper]]. | Group | n | Covariate mean | Raw outcome mean (SD) | Adjusted mean | 95% CI | |---|---|---|---|---|---| | *Example:* standard | 10 | 57.6 | 61.6 (7.89) | 60.30 | [59.28, 61.32] | | *Example:* tutoring | 10 | 54.9 | 63.3 (7.42) | 64.60 | [63.58, 65.62] | | [group] | [n] | [value] | [value] | [value] | [value] | **Results checklist** - [ ] Covariate named, with when it was measured (before treatment) and why it was chosen (pre-specified). - [ ] Slopes check reported (interaction F and p, plus a plot). - [ ] Residual checks (normality, equal variance) summarised. - [ ] Type of sums of squares stated. - [ ] F, df, p and an effect size, with the denominator of any standardised effect named. - [ ] Adjusted means with CIs and the covariate value at which they were computed. - [ ] Pairwise comparisons with adjustment method (three or more groups). - [ ] Raw means shown alongside adjusted means. ## Common mistakes and misconceptions 1. **Adjusting for a covariate measured after treatment.** If tutoring changed study hours, and you adjust for study hours, you remove part of the tutoring effect. Covariates must be fixed before treatment. 2. **"ANCOVA makes non-randomised groups comparable."** It does not. It adjusts only for the measured covariate, only linearly, and with the measurement error the covariate carries. Intact groups that differ on the covariate can give misleading adjusted means (Miller and Chapman 2001). 3. **Skipping the slopes check.** If slopes differ, the "adjusted difference" depends on where you look along the covariate, and a single number is misleading (see Exercise C). 4. **Reporting type I SS with the group entered first.** Then the group test ignores the covariate. Enter the covariate first or use type II. 5. **Reporting raw means as if they were adjusted.** In Example A the raw difference is 1.7 points and the adjusted difference is 4.3. Label both. 6. **Comparing partial η² from ANCOVA with η² from an ANOVA.** The covariate removes residual variance, so partial η² looks much larger. Standardise by the raw outcome SD for comparisons. 7. **Choosing covariates after looking at the results.** Picking the covariate that gives the smallest p-value is a form of p-hacking. Pre-specify covariates in the analysis plan. 8. **Assuming change scores and ANCOVA always agree.** They agree when the slope is close to 1. In randomised trials ANCOVA is generally more precise; in non-randomised comparisons they can disagree sharply (Lord's paradox). ## Pre-post designs and Lord's paradox With a pre-test and a post-test there are three common analyses: post-test only (ANOVA or t-test), change scores (t-test or ANOVA on post minus pre), and ANCOVA (post-test adjusted for pre-test). In a **randomised** study all three estimate the same treatment effect, because randomisation makes the expected baseline difference zero. They differ in precision: post-test only ignores useful information; change scores assume a slope of exactly 1; ANCOVA estimates the slope and is the usual recommendation for randomised trials with one baseline and one follow-up. In a **non-randomised** study the analyses answer different questions and can disagree. Lord (1967) described two statisticians analysing the same data on weight gain in two dining halls with different starting weights. One compared mean change and found no difference between halls; the other ran an ANCOVA and found a difference. Both calculations were correct. The paradox is that the two methods make different assumptions about what would have happened without the "treatment": change scores assume each group would have stayed at its own baseline level apart from a common change, while ANCOVA assumes each group would have regressed toward the common line. No statistical test can tell you which assumption is true; it is a question about the causal process, to be argued from subject knowledge (and today usually expressed with a causal diagram). Practical rule: in randomised trials, use ANCOVA with the baseline as covariate. In observational pre-post comparisons of intact groups, state the causal assumption explicitly, consider both analyses as sensitivity checks, and avoid causal language unless the design supports it. With more than two time points, use a mixed model (stats-mixed-models) or a repeated-measures design (stats-repeated-measures). ## Self-test questions 1. In one sentence, what is an adjusted mean? 2. A group has outcome mean 70, covariate mean 48, grand covariate mean 50 and common slope 0.8. What is its adjusted mean? 3. Why must the covariate be measured before treatment? Give a concrete example of what goes wrong otherwise. 4. What does a significant covariate-by-group interaction mean, and what do you report instead of a single adjusted difference? 5. In R, `anova(lm(post ~ group + pre))` and `anova(lm(post ~ pre + group))` give different F values for group. Which one is the ANCOVA test, and why? 6. Change scores are the special case of ANCOVA with which slope? And a post-test-only analysis? ## My notes and next steps | Item | My entry | |---|---| | Confidence (1 to 5) for computing adjusted means by hand | [ ] | | Confidence (1 to 5) for checking homogeneity of slopes | [ ] | | Confidence (1 to 5) for choosing ANCOVA vs change scores | [ ] | | Open questions | [What I still do not understand] | | Mistakes logged in stats-error-log | [Link or entry numbers] | | Next review date (spacing: 2 days, 1 week, 3 weeks) | YYYY-MM-DD | | Next card to study | [stats-mixed-models / stats-factorial-anova] | ## Sources and further reading Based on standard linear-model methods described in the sources below; all wording, data and examples on this card are original. - Fisher, R. A. (1932). *Statistical Methods for Research Workers* (4th ed.). Oliver and Boyd. - Lord, F. M. (1967). A paradox in the interpretation of group comparisons. *Psychological Bulletin*, 68(5), 304–305. - Searle, S. R., Speed, F. M., and Milliken, G. A. (1980). Population marginal means in the linear model: an alternative to least squares means. *The American Statistician*, 34(4), 216–221. - Miller, G. A., and Chapman, J. P. (2001). Misunderstanding analysis of covariance. *Journal of Abnormal Psychology*, 110(1), 40–48. - Vickers, A. J., and Altman, D. G. (2001). Analysing controlled trials with baseline and follow up measurements. *BMJ*, 323(7321), 1123–1124. - Senn, S. (2006). Change from baseline and analysis of covariance revisited. *Statistics in Medicine*, 25(24), 4334–4344. - Lenth, R. V., and Piaskowski, J. emmeans: Estimated Marginal Means, aka Least-Squares Means (R package, version 2.0.4). - Fox, J., Weisberg, S., and Price, B. car: Companion to Applied Regression (R package, version 3.1-5). - NIST/SEMATECH e-Handbook of Statistical Methods, section 3.2.3 Analysis of Variance Models (ANOVA), and chapter 4 Process Modeling, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain. - Further reading (by name only): de Smith, M. J. (2024). *Statistical Analysis Handbook* (2024 edition, StatsRef); *OpenIntro Statistics*; APA *Publication Manual* (7th ed.) and APA JARS-Quant for reporting standards. ## Answer key **Example B.** 1. *F*(2, 9) = 0.51, *p* = .616. The slopes are similar across ovens, so a common-slope ANCOVA is reasonable. 2. A: $2445 + 10.93 = 2455.9$. B: $2526 - 2.185 \times (1004 - 1000) = 2526 - 8.74 = 2517.3$. (C: $2467 - 2.19 = 2464.8$.) These match the emmeans values 2456, 2517 and 2465. 3. Oven A's loaves had lighter dough than average (995 g), so part of their low volume is due to the dough, not the oven; adjustment moves A up. Oven B had heavier dough (1004 g), so adjustment moves it down. 4. 10645.6 / (10645.6 + 1974.6) = 0.84. 5. B differs from A (adjusted difference 61.3 mL, Holm *p* < .001) and from C (52.4 mL, Holm *p* < .001); A and C do not differ (8.9 mL, Holm *p* = .321). 6. ... *F*(2, 11) = 29.65, *p* < .001, partial η² = .84. Adjusted for dough weight (*M* = 1000 g), oven B produced larger loaves (adjusted *M* = 2517 mL) than ovens A (2456 mL) and C (2465 mL); A and C did not differ. **Exercise C.** Run `anova(lm(score ~ maths * lab, data = x))`: the interaction is large and significant, *F*(1, 12) = 167.02, *p* < .001, so homogeneity of slopes fails and a single adjusted difference is misleading. Slopes (`emtrends`): guided 0.29 lab points per maths point (95% CI [0.21, 0.38]); open 0.99 ([0.90, 1.07]). At maths = 45 the guided format scores higher by 9.76 points (*t*(12) = 10.26, *p* < .001); at maths = 70 the open format scores higher by 7.56 points (*t*(12) = -8.68, *p* < .001). The lines cross where $-40.94 + 0.693 \times \text{maths} = 0$, at maths ≈ 59.1 (from the coefficients: open intercept shift -40.936, slope difference 0.6929). Report: "The effect of lab format depended on maths placement, *F*(1, 12) = 167.02, *p* < .001. Students with low placement scores did better in the guided format (difference at 45: 9.8 points), while students with high scores did better in the open format (difference at 70: 7.6 points)." **Self-test.** 1. The group's mean outcome predicted at a common covariate value (usually the grand mean), using the common slope. 2. $70 - 0.8 \times (48 - 50) = 70 + 1.6 = 71.6$. 3. Because a covariate affected by treatment carries part of the treatment effect, and adjusting for it removes that part. Example: adjusting a tutoring effect for hours studied during the term, when tutoring itself increased study hours. 4. The covariate-outcome slope differs between groups, so the group difference depends on the covariate value. Report the slopes per group and compare groups at several meaningful covariate values (or give the Johnson-Neyman region). 5. `post ~ pre + group`. With type I SS each term is adjusted only for the terms before it; group must come after the covariate to be tested as an adjusted effect. Type II SS avoids the ordering issue. 6. Change scores: slope fixed at 1. Post-test only: slope fixed at 0. ## Revision history | Version | Date | Author | Change | |---|---|---|---| | 0.1 | YYYY-MM-DD | [Name] | First version from template | | [0.2] | YYYY-MM-DD | [Name] | [Replaced example data with own data] |