# Factorial ANOVA Method Card: [Your study question or course] > A method card for two-way between-subjects designs: main effects and the interaction, how to read an interaction > plot, simple effects with emmeans in R and contrasts in statsmodels, partial η² with a confidence interval, and the > type I, II and III sums of squares that make R and Python disagree on unbalanced data. It is for students preparing > for a statistics exam, master's and PhD students running two-factor experiments for a thesis, and analysts testing > two settings at once. It speeds up learning by putting the decisions in order (look at the interaction first, then > decide which effects are meaningful, then choose the sums of squares and the follow-up), by showing one example fully > solved, one half solved and one left for you, and by ending with self-test questions whose answers sit in a separate > key. The worked examples use fictional data; replace them with your own. > Source basis: Fisher (1925, 1935), Yates (1934) on unbalanced designs, Searle (1971) and Langsrud (2003) on sums of > squares, Cohen (1973) on partial η², the NIST/SEMATECH e-Handbook of Statistical Methods, and the R, car, emmeans > and statsmodels documentation, all written up in original words and examples. ## Document control | Field | Value | |---|---| | Course or project | [STAT 201 / thesis chapter 4 / process experiment] | | Learner or analyst | [Your name] | | Data set | [Brightwell Bakery flour x proofing trial (fictional) / your data set and source] | | Software and versions | R 4.5.1 (stats, car 3.1.5, emmeans 2.0.4); Python 3.14 with numpy 2.5.4, pandas 3.1.0, scipy 1.18.1, statsmodels 0.15.0 | | Sums of squares type used | [Type II / Type III with sum-to-zero contrasts / balanced, all types equal] | | Date | YYYY-MM-DD | | Version | [0.1] | ## When to use it **One-sentence rule:** use a factorial (two-way) ANOVA when a continuous outcome is measured on independent units in every combination of two categorical factors, and you want to know whether each factor matters and whether the effect of one factor depends on the level of the other (the interaction). Why not two one-way ANOVAs? Two separate analyses cannot detect an interaction, they waste data (each ignores the variation explained by the other factor, so the error term is larger), and they may mislead when the effect of one factor reverses across levels of the other. A factorial design answers three questions with one experiment: - **Main effect of A:** averaging over B, do the levels of A differ? - **Main effect of B:** averaging over A, do the levels of B differ? - **A x B interaction:** does the effect of A change across the levels of B? On a plot, are the lines not parallel? | Use when | Do not use when | |---|---| | Two categorical factors, each with 2 or more levels, fully crossed (every combination has data) | Some combinations are empty: a fully crossed model cannot be fitted; use a reduced model | | Different units in each cell (between-subjects) | The same units appear in several cells: repeated-measures ANOVA or a mixed model | | Continuous outcome, residuals roughly normal | Counts, proportions or strongly skewed outcomes with small cells: GLM or transformation | | Similar variances across cells | Cell SDs differ a lot: heteroscedasticity-consistent (HC) standard errors or a GLS model | | One factor plus a continuous covariate is not the question | You need to adjust for a continuous covariate: ANCOVA | **Alternatives.** Linear regression with dummy-coded factors and their product gives the identical model and is the easiest route to custom contrasts (academic/stats-linear-regression). Aligned rank transform ANOVA is a nonparametric option for factorial designs. For nested or repeated factors, see academic/stats-mixed-models. ```mermaid flowchart TD A["Fit full model: A + B + A x B"] --> B{"Cells balanced?"} B -->|Yes| C["Types I, II and III agree"] B -->|No| D["Choose SS type before looking: II or III with sum-to-zero contrasts"] C --> E{"Interaction meaningful?"} D --> E E -->|Yes| F["Interpret simple effects: A at each level of B"] E -->|No| G["Interpret main effects, marginal means"] F --> H["Report cell means, interaction plot, partial eta squared"] G --> I["Post hoc on marginal means if a factor has 3+ levels"] I --> H ``` ## The question and the data **Research question (Example A).** Brightwell Bakery (fictional) wants taller loaves. In a trial, 24 loaves were randomly assigned to one of six recipes: two flours (Standard, Wholegrain) crossed with three proofing times (30, 45 and 60 minutes), four loaves per cell. The outcome is loaf height in millimetres. Does height depend on flour, proofing time, and their combination? | Element | Example A | Your study | |---|---|---| | Unit | One loaf | [ ] | | Factor A | Flour: Standard, Wholegrain (2 levels) | [ ] | | Factor B | Proofing time: 30, 45, 60 min (3 levels, treated as categorical) | [ ] | | Outcome | Loaf height, mm (continuous) | [ ] | | Design | 2 x 3 fully crossed, completely randomised, n = 4 per cell, N = 24 (balanced) | [ ] | | Hypotheses | H0 for flour, for proofing time and for the interaction (no effect each); H1: an effect exists | [ ] | Raw data (heights in mm, four loaves per cell): | Flour | 30 min | 45 min | 60 min | |---|---|---|---| | *Example:* Standard | 80, 82, 78, 81 | 88, 90, 86, 89 | 95, 93, 97, 94 | | *Example:* Wholegrain | 72, 75, 70, 73 | 80, 78, 82, 79 | 81, 79, 83, 80 | Cell means and marginal means: | Flour | 30 min | 45 min | 60 min | Flour mean | |---|---|---|---|---| | *Example:* Standard | 80.25 | 88.25 | 94.75 | 87.75 | | *Example:* Wholegrain | 72.50 | 79.75 | 80.75 | 77.67 | | Proofing mean | 76.38 | 84.00 | 87.75 | 82.71 (grand mean) | The interaction plot shows each flour's mean height across proofing times, with 95% CIs based on the pooled error (MS error = 3.15, df = 18). The Standard line keeps rising from 45 to 60 minutes; the Wholegrain line flattens. The lines are not parallel: that is what an interaction looks like. ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "description": "Interaction plot: mean loaf height by proofing time and flour, Brightwell Bakery (fictional data)", "width": 340, "height": 240, "data": {"values": [ {"flour": "Standard", "proof": "30", "mean": 80.25, "lo": 78.38, "hi": 82.12}, {"flour": "Standard", "proof": "45", "mean": 88.25, "lo": 86.38, "hi": 90.12}, {"flour": "Standard", "proof": "60", "mean": 94.75, "lo": 92.88, "hi": 96.62}, {"flour": "Wholegrain", "proof": "30", "mean": 72.5, "lo": 70.63, "hi": 74.37}, {"flour": "Wholegrain", "proof": "45", "mean": 79.75, "lo": 77.88, "hi": 81.62}, {"flour": "Wholegrain", "proof": "60", "mean": 80.75, "lo": 78.88, "hi": 82.62} ]}, "encoding": { "x": {"field": "proof", "type": "ordinal", "title": "Proofing time (min)"}, "color": {"field": "flour", "type": "nominal", "title": "Flour"} }, "layer": [ {"mark": {"type": "line", "point": true}, "encoding": {"y": {"field": "mean", "type": "quantitative", "title": "Loaf height, mm (mean, 95% CI)", "scale": {"domain": [65, 100]}}, "tooltip": [{"field": "flour"}, {"field": "proof"}, {"field": "mean"}, {"field": "lo"}, {"field": "hi"}]}}, {"mark": {"type": "errorbar", "ticks": true}, "encoding": {"y": {"field": "lo", "type": "quantitative"}, "y2": {"field": "hi"}}} ] } ``` ## Assumptions and how to check each | Assumption | How to check | R | Python | If violated | |---|---|---|---|---| | Independent observations | Design: random assignment to cells, one measurement per unit | Design review | Design review | Mixed model (repeated or clustered units) | | Normal residuals | Q-Q plot of residuals from the full model; Shapiro-Wilk as support | `qqnorm(residuals(fit)); shapiro.test(residuals(fit))` | `stats.shapiro(fit.resid)` | Transform the outcome, or use robust or rank-based methods | | Equal variances across cells | Cell SDs (largest / smallest above about 2 is a warning); Levene across all cells | `car::leveneTest(height ~ flour * proof, d)` | `stats.levene(*cells, center="median")` | HC standard errors (`car::Anova(fit, white.adjust = TRUE)`; statsmodels `anova_lm(fit, robust="hc3")`) | | Every cell observed, ideally balanced | `table(d$flour, d$proof)` | `table()` | `pd.crosstab(d.flour, d.proof)` | Unbalanced: choose type II or III sums of squares deliberately | | Fixed factors | The levels are the ones you care about, not a random sample of levels | Design review | Design review | Random factor: mixed model | In Example A the cell SDs are 1.71 in five cells and 2.08 in one (ratio 1.2), Levene's test gives F(5, 18) = 0.04, p = .999, and the residuals look normal (Shapiro-Wilk W = 0.941, p = .167). The design is balanced, so all three types of sums of squares give the same table. ## How it works **Plain language.** Start as if the six cells were six groups in a one-way ANOVA: the variation between cell means is explained variation, the variation within cells is error. A factorial ANOVA splits the between-cells variation into three parts. The **flour** part is how much the flour means (averaged over proofing times) differ. The **proofing** part is how much the proofing-time means (averaged over flours) differ. Whatever between-cell variation is left over is the **interaction**: the part that cannot be explained by adding a flour effect and a proofing effect. If the two effects simply add up, the lines on the plot are parallel and the interaction is near zero. Each part gets its own F test against the same within-cell error. When the interaction is real, a main effect is an average over conditions where the effect differs. "Standard flour is 10 mm taller on average" is true but hides that the gap is 8 mm at short proofing and 14 mm at 60 minutes. That is why you look at the interaction first, then at **simple effects** (the effect of one factor at each level of the other). **Formulas (balanced design).** Factor A with $a$ levels, factor B with $b$ levels, $n$ units per cell, cell means $\bar y_{jk}$, marginal means $\bar y_{j\cdot}$ and $\bar y_{\cdot k}$, grand mean $\bar y$, total $N = abn$: $$ SS_A = bn\sum_{j=1}^{a}(\bar y_{j\cdot}-\bar y)^2, \qquad SS_B = an\sum_{k=1}^{b}(\bar y_{\cdot k}-\bar y)^2 $$ $$ SS_{AB} = n\sum_{j=1}^{a}\sum_{k=1}^{b}(\bar y_{jk}-\bar y_{j\cdot}-\bar y_{\cdot k}+\bar y)^2, \qquad SS_E = \sum_{j,k}\sum_{i=1}^{n}(y_{ijk}-\bar y_{jk})^2 $$ Each effect has $F = MS_{\text{effect}}/MS_E$, with $MS = SS/df$: | Source | df | F | |---|---|---| | A | $a-1$ | $MS_A/MS_E$ | | B | $b-1$ | $MS_B/MS_E$ | | A x B | $(a-1)(b-1)$ | $MS_{AB}/MS_E$ | | Error | $ab(n-1) = N-ab$ | | | Total | $N-1$ | | **Effect size.** Partial η² is the share of variation an effect explains out of that effect plus error, ignoring the other effects: $$ \eta^2_p = \frac{SS_{\text{effect}}}{SS_{\text{effect}}+SS_E} = \frac{F\,df_{\text{effect}}}{F\,df_{\text{effect}}+df_E} $$ Partial η² values for different effects do not add up to 1 and are not comparable across designs with different factors; plain η² (SS_effect / SS_total) is the alternative when you want shares of the total. | Symbol | Meaning | |---|---| | $a$, $b$, $n$, $N$ | Levels of A, levels of B, sample size n per cell, total sample size | | $\bar y_{jk}$ | Mean of cell (level $j$ of A, level $k$ of B) | | $\bar y_{j\cdot}$, $\bar y_{\cdot k}$, $\bar y$ | Marginal mean of A level $j$, of B level $k$, grand mean | | $SS_A$, $SS_B$, $SS_{AB}$, $SS_E$ | Sums of squares for A, B, interaction, error | | $MS_E$ | Mean square error, the pooled within-cell variance | | $\eta^2_p$ | Partial eta squared | ### Types of sums of squares (unbalanced data) When every cell has the same n, the factors are uncorrelated and the formulas above are unambiguous. When cell sizes differ, flour and proofing time become correlated in the data, and "the sum of squares for flour" depends on what else is already in the model. Three conventions exist: - **Type I (sequential):** each term is added in the order you write the formula. Flour first gets all the variation it can claim; proofing then gets what is left. Change the order and the numbers change. R's `anova()` and statsmodels `anova_lm(typ=1)` use type I. - **Type II (hierarchical):** each main effect is adjusted for the other main effect but not for the interaction. It is the most powerful test of main effects when there is no interaction. `car::Anova(fit, type = 2)` and `anova_lm(fit, typ=2)`. - **Type III (marginal):** each term is adjusted for every other term, including the interaction. It tests main effects as unweighted averages over the other factor. It is only meaningful with **sum-to-zero (effect) contrasts**: with R's default treatment contrasts, type III "main effects" test the effect at the reference level of the other factor. SPSS and SAS report type III by default. The interaction term is adjusted for both main effects in all three types, so its SS is identical. To see the differences, the code below drops four loaves from Example A (cell sizes 3, 4, 4 for Standard and 3, 4, 2 for Wholegrain). Run it after the Worked example A code in the same session: it reuses the data frame `d` (and, in Python, the imports) defined there. ```r u <- d[-c(4, 16, 23, 24), ] # unbalanced: drop four rows table(u$flour, u$proof) anova(lm(height ~ flour * proof, data = u)) # type I, flour first anova(lm(height ~ proof * flour, data = u)) # type I, proof first car::Anova(lm(height ~ flour * proof, data = u), type = 2) options(contrasts = c("contr.sum", "contr.poly")) # needed for a meaningful type III car::Anova(lm(height ~ flour * proof, data = u), type = 3) options(contrasts = c("contr.treatment", "contr.poly")) # restore the default ``` ```python import statsmodels.formula.api as smf from statsmodels.stats.anova import anova_lm u = d.drop(index=[3, 15, 22, 23]) # d from Worked example A f_t = smf.ols("height ~ C(flour) * C(proof)", data=u).fit() print(anova_lm(f_t, typ=1).round(2)) print(anova_lm(f_t, typ=2).round(2)) f_s = smf.ols("height ~ C(flour, Sum) * C(proof, Sum)", data=u).fit() print(anova_lm(f_s, typ=3).round(2)) print(anova_lm(f_t, typ=3).round(2)) # default treatment coding: misleading ``` Sums of squares from both programs (identical; R's `car::Anova` prints the type III column rounded to whole numbers): | Term | Type I, flour first | Type I, proof first | Type II | Type III, sum-to-zero | Type III, treatment coding (wrong) | |---|---|---|---|---|---| | Flour | 602.25 | 480.01 | 480.01 | 498.70 | 88.17 | | Proofing time | 442.88 | 565.13 | 442.88 | 385.42 | 373.05 | | Flour x proofing | 42.74 | 42.74 | 42.74 | 42.74 | 42.74 | | Error (df = 14) | 48.92 | 48.92 | 48.92 | 48.92 | 48.92 | Read the table slowly. Type I gives flour 602.25 when it enters first and 480.01 when it enters second. Type II gives each main effect its "entered second" value. Type III differs again because it averages over cells without weighting by cell size. The last column shows the classic error: R's default `contr.treatment` (and statsmodels' default treatment coding) with type III gives flour a sum of squares of only 88.17, because it is then testing the flour difference at 30 minutes only. All analyses agree on the interaction (F(2, 14) = 6.12, p = .012). Which type to report: when the interaction is negligible, type II is usually preferred; when you need main effects in the presence of an interaction (often you should not), type III with sum-to-zero contrasts. Decide in your analysis plan and state it. ## Worked example A (fully solved): flour and proofing time **Hypotheses.** Three tests at α = 0.05: no main effect of flour; no main effect of proofing time; no flour x proofing interaction (the flour difference is the same at every proofing time). **Checks.** Random assignment, one loaf per unit, balanced cells (n = 4). Cell SDs 1.71 to 2.08; Levene p = .999; residuals normal (Shapiro-Wilk p = .167). Balanced, so the type of sums of squares does not matter. **R code and output.** ```r d <- data.frame( flour = factor(rep(c("Standard", "Wholegrain"), each = 12)), proof = factor(rep(rep(c("30", "45", "60"), each = 4), 2)), height = c(80, 82, 78, 81, 88, 90, 86, 89, 95, 93, 97, 94, 72, 75, 70, 73, 80, 78, 82, 79, 81, 79, 83, 80)) with(d, round(tapply(height, list(flour, proof), mean), 2)) fit <- aov(height ~ flour * proof, data = d) summary(fit) shapiro.test(residuals(fit)) # install.packages(c("car", "emmeans")) car::leveneTest(height ~ flour * proof, data = d) tab <- summary(fit)[[1]]; sse <- tab["Residuals", "Sum Sq"] round(tab[1:3, "Sum Sq"] / (tab[1:3, "Sum Sq"] + sse), 3) # partial eta squared library(emmeans) emm <- emmeans(fit, ~ flour | proof) # flour difference at each proofing time confint(pairs(emm)) pairs(emm) ``` ```text (output abridged: R printouts condensed, significance codes removed, labels in parentheses added) 30 45 60 Standard 80.25 88.25 94.75 Wholegrain 72.50 79.75 80.75 Df Sum Sq Mean Sq F value Pr(>F) flour 1 610.0 610.0 193.493 4.52e-11 *** proof 2 537.6 268.8 85.256 6.60e-10 *** flour:proof 2 46.6 23.3 7.388 0.00454 ** Residuals 18 56.7 3.2 Shapiro-Wilk normality test: W = 0.9405, p-value = 0.1673 Levene's Test (center = median): group Df 5, F value 0.0423, Pr(>F) 0.9988; residual Df 18 [1] 0.915 0.905 0.451 (partial eta squared: flour, proof, flour:proof) proof = 30: contrast estimate SE df lower.CL upper.CL Standard - Wholegrain 7.75 1.26 18 5.11 10.4 proof = 45: Standard - Wholegrain 8.50 1.26 18 5.86 11.1 proof = 60: Standard - Wholegrain 14.00 1.26 18 11.36 16.6 proof = 30: Standard - Wholegrain t.ratio 6.173 p.value <0.0001 proof = 45: Standard - Wholegrain t.ratio 6.770 p.value <0.0001 proof = 60: Standard - Wholegrain t.ratio 11.151 p.value <0.0001 ``` 95% CI for partial η² of the interaction, by inverting the noncentral F (Steiger 2004; N for the conversion is df_effect + df_error + 1 = 21). This continues the R session above (it uses `tab`): ```r Fobs <- tab["flour:proof", "F value"] f <- function(ncp) pf(Fobs, 2, 18, ncp) lo <- uniroot(function(n) f(n) - 0.975, c(0, 100))$root hi <- uniroot(function(n) f(n) - 0.025, c(0, 200))$root round(c(lo / (lo + 21), hi / (hi + 21)), 3) ``` ```text [1] 0.067 0.634 ``` **Python code and output.** ```python import pandas as pd import statsmodels.formula.api as smf from statsmodels.stats.anova import anova_lm d = pd.DataFrame({ "flour": ["Standard"] * 12 + ["Wholegrain"] * 12, "proof": (["30"] * 4 + ["45"] * 4 + ["60"] * 4) * 2, "height": [80, 82, 78, 81, 88, 90, 86, 89, 95, 93, 97, 94, 72, 75, 70, 73, 80, 78, 82, 79, 81, 79, 83, 80]}) print(d.pivot_table(index="flour", columns="proof", values="height", aggfunc="mean")) fit = smf.ols("height ~ C(flour) * C(proof)", data=d).fit() tab = anova_lm(fit, typ=2) # balanced: types 1, 2 and 3 agree print(tab.round(4)) sse = tab.loc["Residual", "sum_sq"] print((tab.sum_sq[:-1] / (tab.sum_sq[:-1] + sse)).round(3)) # partial eta squared # Simple effects of flour at each proofing time (pooled error, df = 18) b = "C(flour)[T.Wholegrain]" for p, c in [("30", f"{b} = 0"), ("45", f"{b} + {b}:C(proof)[T.45] = 0"), ("60", f"{b} + {b}:C(proof)[T.60] = 0")]: r = fit.t_test(c) print(p, "Wholegrain - Standard:", round(float(r.effect[0]), 2), "SE", round(float(r.sd[0, 0]), 3), "t", round(float(r.tvalue[0, 0]), 3), "p", f"{float(r.pvalue):.2e}") ``` ```text (output abridged: the "Name: sum_sq, dtype: float64" footer removed) proof 30 45 60 flour Standard 80.25 88.25 94.75 Wholegrain 72.50 79.75 80.75 sum_sq df F PR(>F) C(flour) 610.0417 1.0 193.4934 0.0000 C(proof) 537.5833 2.0 85.2555 0.0000 C(flour):C(proof) 46.5833 2.0 7.3877 0.0045 Residual 56.7500 18.0 NaN NaN C(flour) 0.915 C(proof) 0.905 C(flour):C(proof) 0.451 30 Wholegrain - Standard: -7.75 SE 1.256 t -6.173 p 7.93e-06 45 Wholegrain - Standard: -8.5 SE 1.256 t -6.77 p 2.42e-06 60 Wholegrain - Standard: -14.0 SE 1.256 t -11.151 p 1.63e-09 ``` statsmodels has no emmeans equivalent built in, so simple effects are written as linear combinations of the treatment-coded coefficients: with Standard and 30 minutes as reference levels, the flour difference at 45 minutes is the flour coefficient plus the flour x 45 interaction coefficient. The results match emmeans exactly, with the sign reversed because the Python contrast is Wholegrain − Standard. The noncentral-F CI for partial η² can be computed in Python with the same `eta2_ci` function shown in academic/stats-one-way-anova (`eta2_ci(7.387665, 2, 18)` returns (0.067, 0.634)). **95% or 90%?** The interval above is a two-sided 95% interval (2.5% in each tail). Steiger (2004) shows that a **90%** interval matches the F test for measures such as (partial) η², because η² cannot be negative and the F test is one-sided: a 90% interval excludes 0 exactly when the F test is significant at α = 0.05. Replacing 0.975 and 0.025 with 0.95 and 0.05 in the code gives a 90% CI of [0.112, 0.603] for the interaction. Both are used in published work; state the level, and keep it the same for every effect you report. **By hand.** Grand mean = 496.25 / 6 = 82.708. - $SS_{\text{flour}} = 3 \times 4 \times [(87.75-82.708)^2 + (77.667-82.708)^2] = 12(25.42 + 25.42) = 610.04$, df = 1. - $SS_{\text{proof}} = 2 \times 4 \times [(76.375-82.708)^2 + (84.0-82.708)^2 + (87.75-82.708)^2] = 8(40.11 + 1.67 + 25.42) = 537.58$, df = 2. - $SS_{\text{cells}} = 4 \sum (\bar y_{jk} - 82.708)^2 = 4(6.04 + 30.71 + 145.00 + 104.21 + 8.75 + 3.84) = 1194.21$, so $SS_{\text{interaction}} = 1194.21 - 610.04 - 537.58 = 46.58$, df = 2. - $SS_E = 3 \times (2.917 \times 5 + 4.333) = 56.75$, df = 18, $MS_E = 3.153$. - $F_{\text{interaction}} = (46.58/2) / 3.153 = 7.39$; partial η² = 46.58 / (46.58 + 56.75) = 0.451. **Effect sizes with CIs.** Partial η²: flour 0.915, proofing 0.905, interaction 0.451 (95% CI [0.067, 0.634]). The more useful effect sizes are the simple-effect mean differences: Standard flour is 7.75 mm taller at 30 minutes (95% CI [5.11, 10.4]), 8.50 mm at 45 minutes ([5.86, 11.1]) and 14.00 mm at 60 minutes ([11.36, 16.6]). **Interpretation.** Both factors matter, but not additively. Standard flour gives taller loaves at every proofing time, and longer proofing helps both flours, but the extra 15 minutes from 45 to 60 adds 6.5 mm for Standard and only 1.0 mm for Wholegrain. The interaction (F(2, 18) = 7.39, p = .005) captures exactly this: the flour gap grows from about 8 mm to 14 mm at 60 minutes. A practical reading is that 60-minute proofing pays off for Standard dough but hardly for Wholegrain. The large partial η² values reflect a tightly controlled trial with little within-cell noise; they would shrink in a noisier setting, which is why the millimetre differences are the better summary. **APA-style report.** A 2 (flour) x 3 (proofing time) between-subjects ANOVA on loaf height showed main effects of flour, *F*(1, 18) = 193.49, *p* < .001, η²p = .92, and proofing time, *F*(2, 18) = 85.26, *p* < .001, η²p = .90, qualified by a flour x proofing interaction, *F*(2, 18) = 7.39, *p* = .005, η²p = .45, 95% CI [.07, .63]. Simple effects showed that Standard flour produced taller loaves than Wholegrain at 30 min (7.75 mm, 95% CI [5.11, 10.39]), 45 min (8.50 mm, [5.86, 11.14]) and 60 min (14.00 mm, [11.36, 16.64]), all *p* < .001; the advantage was largest at 60 min because Wholegrain loaves gained little from the extra proofing. ## Worked example B (half solved): noise and task type **Question.** Sixteen Lakemoor University (fictional) volunteers were randomly assigned to a quiet or a noisy room and to a reading or an arithmetic task (four per cell). The outcome is a score out of 20. | Noise | Reading | Arithmetic | |---|---|---| | *Example:* Quiet | 16, 15, 17, 16 | 14, 15, 13, 14 | | *Example:* Noisy | 11, 12, 10, 11 | 13, 14, 12, 13 | ```r b <- data.frame(noise = factor(rep(c("Quiet", "Noisy"), each = 8), levels = c("Quiet", "Noisy")), task = factor(rep(rep(c("Reading", "Arithmetic"), each = 4), 2), levels = c("Reading", "Arithmetic")), score = c(16, 15, 17, 16, 14, 15, 13, 14, 11, 12, 10, 11, 13, 14, 12, 13)) fb <- aov(score ~ noise * task, b); summary(fb) library(emmeans); pairs(emmeans(fb, ~ noise | task)) ``` ```text (output abridged: cell-means table added for reference, R printouts condensed) Reading Arithmetic (cell means) Quiet 16 14 Noisy 11 13 Df Sum Sq Mean Sq F value Pr(>F) noise 1 36 36.00 54 8.87e-06 *** task 1 0 0.00 0 1.000000 noise:task 1 16 16.00 24 0.000367 *** Residuals 12 8 0.67 task = Reading: Quiet - Noisy estimate 5 SE 0.577 df 12 t.ratio 8.660 p.value <0.0001 task = Arithmetic: Quiet - Noisy estimate 1 SE 0.577 df 12 t.ratio 1.732 p.value 0.1089 ``` The Python version (`smf.ols("score ~ C(noise) * C(task)", data=b)` with `anova_lm(fit, typ=2)`) gives the same table. 1. Marginal means: Quiet = [ ], Noisy = [ ], Reading = [ ], Arithmetic = [ ]. 2. Why is the task main effect exactly zero? [ ] 3. Check the interaction SS by hand: cell means minus additive predictions, squared, times n: [ ] (should be 16). 4. Partial η²: noise = 36 / (36 + 8) = [ ]; interaction = [ ]. 5. Simple effects: in which task does noise matter, and how much? [ ] 6. Would it be correct to write "task had no effect on scores"? Explain: [ ] 7. Report sentence: [ ] ## Exercise C (unsolved): oven temperature and sugar level Brightwell Bakery (fictional) tested two oven temperatures (Low, High) and two sugar levels (Low, High) on a crust colour score, five pastries per cell (balanced, N = 20). The error sum of squares is $SS_E = 64$. | Temperature | Sugar Low | Sugar High | |---|---|---| | *Example:* Low | 20 | 22 | | *Example:* High | 24 | 30 | Tasks: 1. Sketch the interaction plot. Are the lines parallel? 2. Compute the grand mean, the marginal means and $SS_{\text{temperature}}$, $SS_{\text{sugar}}$, $SS_{\text{interaction}}$. 3. Build the full ANOVA table (df, MS, F, p). Use `pf(F, 1, 16, lower.tail = FALSE)` or `stats.f.sf(F, 1, 16)`. 4. Compute partial η² for each effect. 5. Describe the simple effect of sugar at each temperature in words, and write the APA sentence for the interaction. ## Reading software output | Output line | What it means | What to report | |---|---|---| | `flour Df 1 Sum Sq 610.0 F 193.493` | Main effect of flour, averaged over proofing times | *F*(1, 18) = 193.49, *p* < .001, η²p | | `proof Df 2 Sum Sq 537.6 F 85.256` | Main effect of proofing time, averaged over flours | *F*(2, 18) = 85.26, *p* < .001, η²p | | `flour:proof Df 2 Sum Sq 46.6 F 7.388` | Interaction: does the flour effect change across proofing times? | Read first; *F*(2, 18) = 7.39, *p* = .005 | | `Residuals Df 18 Mean Sq 3.2` | Within-cell error, MS_E; error df = N − ab | Second df of every F | | R `anova(lm(...))` | Type I (sequential) sums of squares | Fine if balanced; on unbalanced data state the order or switch type | | `Anova Table (Type II tests)` / `(Type III tests)` | car's adjusted sums of squares | Name the type; for type III confirm sum-to-zero contrasts | | statsmodels `anova_lm(typ=1/2/3)` | Same three types; type 3 needs `C(x, Sum)` coding | Name the type | | emmeans `contrast ... estimate SE df t.ratio p.value` | Simple effect: difference at one level of the other factor, pooled error | Difference with 95% CI and p | | `(Intercept)` row in type III tables | Test that the grand mean is zero | Never report | ## Reporting template **Fill-in sentence.** A [a] ([factor A]) x [b] ([factor B]) between-subjects ANOVA on [outcome] showed [a main effect of A, *F*([df], [df]) = [ ], *p* = [ ], η²p = [ ]], [a main effect of B, ...], [and / qualified by] a [factor A x factor B] interaction, *F*([df], [df]) = [ ], *p* = [ ], η²p = [ ], 95% CI [[lo], [hi]]. Simple effects showed that [the effect of A was [size] at [level of B] and [size] at [level of B]]. [Type II / type III, sum-to-zero] sums of squares were used [because cell sizes were unequal]. | Factor A level | B level 1 M (SD) | B level 2 M (SD) | B level 3 M (SD) | n per cell | |---|---|---|---|---| | *Example:* Standard | 80.25 (1.71) | 88.25 (1.71) | 94.75 (1.71) | 4 | | *Example:* Wholegrain | 72.50 (2.08) | 79.75 (1.71) | 80.75 (1.71) | 4 | | [Level] | [ ] | [ ] | [ ] | [ ] | Results checklist: - [ ] Design described: factors, levels, between-subjects, cell sizes - [ ] Cell means and SDs in a table, and an interaction plot with CIs - [ ] Assumption checks summarised (residuals, cell SDs) - [ ] Sums-of-squares type stated (and contrasts for type III), or "balanced design" - [ ] Interaction reported first; main effects interpreted only in its light - [ ] Simple effects or interaction contrasts with mean differences and 95% CIs - [ ] Partial η² (or η² or ω²) named precisely, with a CI for the key effect - [ ] Multiple simple-effect tests adjusted if there are many ## Common mistakes and misconceptions 1. **Interpreting main effects while ignoring a strong interaction.** In Example B, "task had no effect" is false: task matters in opposite directions in the two rooms. 2. **Reading R's `anova()` on unbalanced data as if order did not matter.** It is type I; swapping the order of factors changed the flour SS from 602.25 to 480.01. 3. **Type III with treatment contrasts.** `car::Anova(type = 3)` on a model fitted with R's default contrasts tests main effects at the reference level (flour SS 88.17 instead of 498.70). Set `contr.sum` first, or use type II. 4. **Comparing "significant here, not significant there".** That the noise effect is significant for reading and not for arithmetic is not itself a test of the interaction; the interaction F is. 5. **Running separate one-way ANOVAs or t-tests per level.** Each uses its own small error term and loses the interaction test; use simple effects with the pooled error from the full model. 6. **Adding partial η² values and calling the total the variance explained.** Partial η² values overlap; they can sum to more than 1 (0.915 + 0.905 + 0.451 here). 7. **Treating a numeric factor as categorical without thinking.** With three proofing times, categorical is fine; with many levels, a regression with a trend term may be more informative. 8. **"Non-parallel lines on the plot prove an interaction."** Small samples produce non-parallel lines by chance; look at the CIs and the interaction test. 9. **Forgetting that the same unit must appear in only one cell.** If each loaf were measured twice, the design would need a repeated-measures or mixed model. ## Self-test questions 1. A 3 x 4 between-subjects design has 5 units per cell. Give the df for both main effects, the interaction and error. 2. Cell means are: A1B1 = 10, A1B2 = 14, A2B1 = 12, A2B2 = 16. Is there an interaction? Explain from the means. 3. Your cells have sizes 12, 9, 15 and 6 and the interaction is negligible. Which sums-of-squares type do you choose, and what must you state in the write-up? 4. Why does type III require sum-to-zero contrasts to give meaningful main-effect tests? 5. Partial η² for an interaction is 0.45, but the interaction explains only 3.7% of the total sum of squares. How can both be true? 6. What is a simple effect, and why does emmeans use the pooled error from the full model rather than the data from one level only? ## My notes and next steps | Item | My entry | |---|---| | Confidence (1-5) in reading an interaction plot | [ ] | | Confidence (1-5) in choosing type I, II or III sums of squares | [ ] | | Confidence (1-5) in running and reporting simple effects | [ ] | | What I still find confusing | [ ] | | Questions for my tutor or supervisor | [ ] | | Mistakes logged in academic/stats-error-log | [Link or entry IDs] | | Next review date (spacing) | YYYY-MM-DD | | Next template to study | [academic/stats-repeated-measures or academic/stats-ancova] | ## Sources and further reading Structure aligned with the method-card shape of this statistics library; methods based on the original papers and the public-domain NIST/SEMATECH e-Handbook, written in original words with fictional examples. - Fisher, R. A. (1925). *Statistical Methods for Research Workers*. Oliver and Boyd. - Fisher, R. A. (1935). *The Design of Experiments*. Oliver and Boyd. - Yates, F. (1934). The analysis of multiple classifications with unequal numbers in the different classes. *Journal of the American Statistical Association*, 29(185), 51–66. - Searle, S. R. (1971). *Linear Models*. Wiley. - Langsrud, Ø. (2003). ANOVA for unbalanced data: use Type II instead of Type III sums of squares. *Statistics and Computing*, 13(2), 163–167. - Cohen, J. (1973). Eta-squared and partial eta-squared in fixed factor ANOVA designs. *Educational and Psychological Measurement*, 33(1), 107–112. - Steiger, J. H. (2004). Beyond the F test: effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. *Psychological Methods*, 9(2), 164–182. - Fox, J., and Weisberg, S. (2019). *An R Companion to Applied Regression* (3rd ed.). Sage; and the car package documentation for `Anova` (3.1-5). - Lenth, R. V. emmeans package documentation (2.0.4). - NIST/SEMATECH e-Handbook of Statistical Methods, sections 7.4.3.7 The two-way ANOVA and 5.3.3.3 Full factorial designs, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain. - R Core Team. Documentation for `aov`, `anova`, `lm`, `contr.sum` (R 4.5.1); statsmodels documentation for `anova_lm` and `OLSResults.t_test` (0.15.0). - Further reading (by name only): de Smith, M. J. (2024). *Statistical Analysis Handbook*, 2024 edition. StatsRef, https://www.statsref.com/; OpenIntro Statistics; APA *Publication Manual* (7th ed.) and APA JARS-Quant. ## Answer key **Worked example B.** 1. Quiet = 15, Noisy = 12, Reading = 13.5, Arithmetic = 13.5 (grand mean 13.5). 2. Averaged over rooms, reading and arithmetic scores are equal (13.5 each): reading is better in the quiet room and worse in the noisy room, and the two differences cancel. 3. Additive prediction for Quiet-Reading = 15 + 13.5 − 13.5 = 15; the cell mean is 16, a deviation of +1. All four deviations are ±1, so $SS_{AB} = 4 \times (1 + 1 + 1 + 1) = 16$. 4. Noise: 36 / 44 = 0.818; interaction: 16 / 24 = 0.667; task: 0. 5. Noise lowers reading scores by 5 points (t(12) = 8.66, p < .001) but arithmetic scores by only 1 point, which is not detectable (t(12) = 1.73, p = .109). 6. No. Task had no effect averaged over rooms, but it mattered within each room in opposite directions; the interaction shows that the task effect depends on noise. 7. A 2 x 2 ANOVA showed a main effect of noise, *F*(1, 12) = 54.00, *p* < .001, η²p = .82, no main effect of task, *F*(1, 12) = 0.00, *p* = 1.000, and a noise x task interaction, *F*(1, 12) = 24.00, *p* < .001, η²p = .67: noise reduced reading scores by 5 points but arithmetic scores by only 1 point (*p* = .109). **Exercise C.** The lines are not parallel (sugar adds 2 at Low temperature and 6 at High). Grand mean = 24. Temperature means 21 and 27; sugar means 22 and 26. $SS_{\text{temp}} = 10[(21-24)^2 + (27-24)^2] = 180$; $SS_{\text{sugar}} = 10[(22-24)^2 + (26-24)^2] = 80$; $SS_{\text{cells}} = 5(16 + 4 + 0 + 36) = 280$, so $SS_{\text{int}} = 280 - 180 - 80 = 20$. $MS_E = 64 / 16 = 4$. Temperature: F(1, 16) = 45.0, *p* < .001, η²p = .74. Sugar: F(1, 16) = 20.0, *p* < .001 (0.000385), η²p = .56. Interaction: F(1, 16) = 5.0, *p* = .040, η²p = .24. High sugar raises the score by 2 points at low temperature and by 6 points at high temperature. Sentence: The temperature x sugar interaction was significant, *F*(1, 16) = 5.00, *p* = .040, η²p = .24: the benefit of high sugar was three times larger at high oven temperature. **Self-test.** 1. A: 2; B: 3; A x B: 6; error: 12 x (5 − 1) = 48. 2. No. The B effect is +4 at both A levels (and the A effect is +2 at both B levels): the lines are parallel. 3. Type II (or type III with sum-to-zero contrasts if your field or plan requires it). State the type, and for type III the contrasts, because the main-effect numbers depend on it. 4. Type III tests each main effect with the interaction in the model. With treatment coding, the main-effect coefficients are differences at the reference level of the other factor, so the test asks a narrow question. With sum-to-zero coding they are deviations from unweighted averages, which is the intended main-effect question. 5. Partial η² divides by SS_effect + SS_E only: 46.58 / (46.58 + 56.75) = 0.45. Plain η² divides by the total sum of squares, which includes the large main effects: 46.58 / 1250.96 = 0.037. 6. A simple effect is the effect of one factor at a single level of the other. Using the pooled error from all cells gives more df and a more precise error estimate than analysing one level alone, provided variances are similar. ## Revision history | Version | Date | Author | Change | |---|---|---|---| | 0.1 | YYYY-MM-DD | [Your name] | First version from the template | | [0.2] | YYYY-MM-DD | [ ] | [Replaced example data with my own] |