# One-Way ANOVA Method Card: [Your study question or course] > A method card for comparing the means of three or more independent groups: the logic of between-group versus > within-group variance, the F statistic and the ANOVA table by hand, the assumptions and their checks, Welch's ANOVA > for unequal variances, post hoc comparisons (Tukey HSD and Games-Howell), planned contrasts, and the effect sizes η² > and ω² with a confidence interval. It is for students preparing for a statistics exam, master's and PhD students > comparing conditions in thesis data, and analysts comparing several groups at once. It speeds up learning by putting > the decisions in order (classic or Welch, which follow-up, which effect size, which sentence), by showing one example > fully solved, one half solved and one left for you, and by ending with self-test questions whose answers sit in a > separate key. The worked examples use fictional data; replace them with your own. > Source basis: Fisher (1925), Welch (1951), Tukey (1949, 1953), Games and Howell (1976), Hays (1963) for ω², Cohen > (1988), the NIST/SEMATECH e-Handbook of Statistical Methods, and the R, SciPy and statsmodels documentation, all > written up in original words and examples. ## Document control | Field | Value | |---|---| | Course or project | [STAT 101 / thesis chapter 4 / client project] | | Learner or analyst | [Your name] | | Data set | [Lakemoor University study-method experiment (fictional) / your data set and source] | | Software and versions | R 4.5.1 (stats, car 3.1.5); Python 3.14 with numpy 2.5.4, pandas 3.1.0, scipy 1.18.1, statsmodels 0.15.0 | | Date | YYYY-MM-DD | | Version | [0.1] | ## When to use it **One-sentence rule:** use a one-way ANOVA when you want to know whether the mean of a continuous outcome differs across three or more independent groups defined by one factor, and you want one overall test before comparing pairs. Why not just run t-tests on every pair? With three groups there are three pairs, with five groups ten. Each t-test at α = 0.05 has a 5% false-positive risk, so the chance of at least one false positive across all pairs grows quickly (about 14% for three independent tests). ANOVA tests all means at once with a single α, and post hoc procedures then control the family-wise error rate across the pairs. | Use when | Do not use when | |---|---| | One factor with 3 or more levels (for 2 levels, a Welch t-test is simpler and equivalent to Welch ANOVA) | Two or more factors: use factorial ANOVA | | Groups are independent (different units in each group) | The same units are measured in each condition: repeated-measures ANOVA or a mixed model | | Outcome is continuous (interval or ratio), or a score with many values | Outcome is a count, a proportion or ordinal with few levels: use a GLM or a nonparametric test | | Residuals roughly normal, or groups large enough (about 15 or more per group, a rule of thumb) | Strong skew or outliers in small groups: Kruskal-Wallis, a transformation, or a robust ANOVA | | You want to compare means | You want to adjust for a continuous covariate: ANCOVA | **Classic or Welch?** The classic F test assumes equal population variances. Welch's ANOVA does not, and it costs almost nothing when variances are in fact equal. A sensible default, matching the Welch-by-default rule for t-tests, is: report Welch's ANOVA with Games-Howell comparisons unless equal variances are justified by design; report the classic ANOVA with Tukey HSD when the group SDs are similar and group sizes are equal. Do not choose between them on the basis of a variance test's p-value alone. **Alternatives.** Kruskal-Wallis with Dunn's test for ranks (academic/stats-nonparametric-tests); a permutation F test (academic/stats-resampling-methods); linear regression with a categorical predictor, which gives the identical F (academic/stats-linear-regression). ```mermaid flowchart TD A["Continuous outcome, one factor, 3+ independent groups"] --> B{"Group SDs similar and sizes equal?"} B -->|Yes| C["Classic ANOVA F test"] B -->|"No, or unsure"| D["Welch ANOVA"] C --> E{"Specific hypotheses stated in advance?"} D --> E E -->|Yes| F["Planned contrasts"] E -->|No| G{"Equal variances?"} G -->|Yes| H["Tukey HSD"] G -->|No| I["Games-Howell"] F --> J["Report means, CIs, F, df, p, omega squared"] H --> J I --> J A --> K{"Strong skew or outliers in small groups?"} K -->|Yes| L["Kruskal-Wallis or transform"] K -->|No| B ``` ## The question and the data **Research question (Example A).** At Lakemoor University (fictional), 24 STAT 101 volunteers were randomly assigned to one of three revision methods for one week: rereading notes, practice tests, or spaced flashcards. Does the mean score on a 100-point quiz differ by method? | Element | Example A | Your study | |---|---|---| | Unit | One student | [ ] | | Factor | Revision method: Reread, Practice, Flashcard (nominal, 3 levels) | [ ] | | Outcome | Quiz score, 0 to 100 (treated as continuous) | [ ] | | Design | Completely randomised, n = 8 per group, N = 24 | [ ] | | Hypotheses | H0: μ_Reread = μ_Practice = μ_Flashcard. H1: at least one mean differs | [ ] | | Method | Scores | n | Mean | SD | |---|---|---|---|---| | *Example:* Reread | 58, 62, 55, 64, 60, 57, 61, 59 | 8 | 59.50 | 2.88 | | *Example:* Practice | 70, 66, 74, 63, 78, 69, 72, 68 | 8 | 70.00 | 4.69 | | *Example:* Flashcard | 65, 72, 58, 80, 61, 75, 69, 64 | 8 | 68.00 | 7.41 | | All | | 24 | 65.83 | | The group means with their 95% confidence intervals (each from its own group SD) show the pattern before any test. Notice that the Flashcard interval is much wider: its SD is about 2.6 times the Reread SD, a warning about the equal-variance assumption. ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "description": "Mean quiz score with 95% CI by revision method, Lakemoor University (fictional data)", "width": 320, "height": 220, "data": {"values": [ {"method": "Reread", "mean": 59.5, "lo": 57.09, "hi": 61.91}, {"method": "Practice", "mean": 70.0, "lo": 66.08, "hi": 73.92}, {"method": "Flashcard", "mean": 68.0, "lo": 61.81, "hi": 74.19} ]}, "encoding": { "x": {"field": "method", "type": "nominal", "title": "Revision method", "sort": ["Reread", "Practice", "Flashcard"]} }, "layer": [ {"mark": {"type": "errorbar", "ticks": true}, "encoding": {"y": {"field": "lo", "type": "quantitative", "title": "Quiz score (mean, 95% CI)", "scale": {"domain": [50, 80]}}, "y2": {"field": "hi"}}}, {"mark": {"type": "point", "filled": true, "size": 80}, "encoding": {"y": {"field": "mean", "type": "quantitative"}, "tooltip": [{"field": "method"}, {"field": "mean"}, {"field": "lo"}, {"field": "hi"}]}} ] } ``` ## Assumptions and how to check each | Assumption | How to check | R | Python | If violated | |---|---|---|---|---| | Independent observations | Design: random assignment, one score per unit, no clustering | Design review | Design review | Repeated measures or mixed model | | Normal residuals (within each group) | Q-Q plot of residuals; Shapiro-Wilk as a supporting check | `qqnorm(residuals(fit)); shapiro.test(residuals(fit))` | `stats.probplot(fit.resid, plot=plt); stats.shapiro(fit.resid)` | Large groups: usually fine (CLT). Small groups with skew: transform or Kruskal-Wallis | | Equal variances | Compare group SDs (largest / smallest above about 2 is a warning); Levene (Brown-Forsythe, median-centred) | `car::leveneTest(score ~ method, d)` | `stats.levene(*groups, center="median")` | Welch ANOVA plus Games-Howell | | No influential outliers | Boxplots per group; standardised residuals beyond ±3 | `boxplot(score ~ method, d)` | `d.boxplot("score", by="method")` | Check for data errors; report with and without; robust methods | | Groups defined before seeing data | Analysis plan | Plan document | Plan document | Label post hoc groupings as exploratory | In Example A the residuals look normal (Shapiro-Wilk W = 0.989, p = .994), but the SD ratio is 7.41 / 2.88 = 2.6 and Levene's median-centred test gives F(2, 21) = 3.41, p = .052. The p-value is just above 0.05, which shows why you should not let a variance test decide for you: the SD ratio alone justifies Welch's ANOVA here. `car::leveneTest` centres on the median by default; `scipy.stats.levene` also defaults to `center="median"`, so both report the same Brown-Forsythe version. ## How it works **Plain language.** Every score differs from the grand mean for two reasons: its group's mean differs from the grand mean (the **between-group** part, which the factor might explain), and the score differs from its own group's mean (the **within-group** part, which is noise). ANOVA splits the total variation into these two pieces. If H0 is true, the group means differ only because of noise, so the between-group variation per degree of freedom should be about the same size as the within-group variation per degree of freedom. Their ratio, F, should be near 1. If the group means really differ, the between part is inflated and F is large. **Formulas.** With $k$ groups, $n_j$ units in group $j$, group means $\bar y_j$, grand mean $\bar y$ and total sample size $N$: $$ SS_B = \sum_{j=1}^{k} n_j(\bar y_j-\bar y)^2, \qquad SS_W = \sum_{j=1}^{k}\sum_{i=1}^{n_j}(y_{ij}-\bar y_j)^2 = \sum_{j=1}^{k}(n_j-1)s_j^2 $$ $$ MS_B = \frac{SS_B}{k-1}, \qquad MS_W = \frac{SS_W}{N-k}, \qquad F = \frac{MS_B}{MS_W} \sim F_{k-1,\,N-k} \text{ under } H_0 $$ The ANOVA table: | Source | SS | df | MS | F | p | |---|---|---|---|---|---| | Between groups (factor) | $SS_B$ | $k-1$ | $SS_B/(k-1)$ | $MS_B/MS_W$ | from $F_{k-1,N-k}$ | | Within groups (residual) | $SS_W$ | $N-k$ | $SS_W/(N-k)$ | | | | Total | $SS_T = SS_B + SS_W$ | $N-1$ | | | | **Welch's ANOVA** weights each group by $w_j = n_j/s_j^2$, so precise groups count more, and replaces the denominator df with an estimate based on the group variances (Welch 1951). Its F and df are not the classic ones; read them from the output. **Effect sizes.** η² is the share of total variation explained by the factor in this sample; it is biased upwards. ω² corrects that bias and estimates the population share: $$ \eta^2 = \frac{SS_B}{SS_T}, \qquad \omega^2 = \frac{SS_B-(k-1)MS_W}{SS_T+MS_W} $$ **Planned contrast.** A contrast is a weighted sum of means with weights $c_j$ that add to zero, tested with the pooled error: $$ \hat L = \sum_j c_j \bar y_j, \qquad SE(\hat L) = \sqrt{MS_W \sum_j \frac{c_j^2}{n_j}}, \qquad t = \frac{\hat L}{SE(\hat L)} \text{ with } df = N-k $$ **Tukey HSD** compares every pair using the studentised range distribution, which allows for the fact that the largest and smallest of $k$ means are compared. With unequal group sizes the Tukey-Kramer version (Kramer 1956) is used, which R's `TukeyHSD`, statsmodels' `pairwise_tukeyhsd` and SciPy's `tukey_hsd` apply automatically. **Games-Howell** does the same with Welch-type standard errors and df for each pair, so it does not assume equal variances. | Symbol | Meaning | |---|---| | $k$, $n_j$, $N$ | Number of groups, sample size n in group $j$, total sample size | | $\bar y_j$, $\bar y$, $s_j$ | Group mean, grand mean, group SD | | $SS_B$, $SS_W$, $SS_T$ | Between-group, within-group, total sum of squares | | $MS_B$, $MS_W$ | Mean squares (SS divided by df); $MS_W$ is the pooled variance | | $F$ | Ratio of mean squares, with df $(k-1, N-k)$ | | $\eta^2$, $\omega^2$ | Proportion of variance explained (sample, bias-corrected) | | $c_j$, $\hat L$ | Contrast weights, contrast estimate | ## Worked example A (fully solved): revision methods **Hypotheses.** H0: the three population mean quiz scores are equal. H1: at least one differs. α = 0.05. Planned contrast (stated before data collection): the two active methods (Practice, Flashcard) versus Reread, weights (−1, 0.5, 0.5). **Checks.** Random assignment, one score per student: independence holds. Residuals are close to normal (Shapiro-Wilk p = .994). Group SDs range from 2.88 to 7.41 (ratio 2.6), so we report the classic ANOVA for learning and Welch's ANOVA with Games-Howell as the primary analysis. **R code and output.** ```r d <- data.frame( method = factor(rep(c("Reread", "Practice", "Flashcard"), each = 8), levels = c("Reread", "Practice", "Flashcard")), score = c(58, 62, 55, 64, 60, 57, 61, 59, 70, 66, 74, 63, 78, 69, 72, 68, 65, 72, 58, 80, 61, 75, 69, 64)) fit <- aov(score ~ method, data = d) summary(fit) # classic ANOVA oneway.test(score ~ method, data = d) # Welch ANOVA: var.equal = FALSE is the default shapiro.test(residuals(fit)) # install.packages("car") car::leveneTest(score ~ method, data = d) # median-centred (Brown-Forsythe) TukeyHSD(fit) ``` ```text (output abridged: headers, significance codes and blank lines removed) Df Sum Sq Mean Sq F value Pr(>F) method 2 497.3 248.67 8.762 0.00171 ** Residuals 21 596.0 28.38 One-way analysis of means (not assuming equal variances) F = 15.796, num df = 2.000, denom df = 12.452, p-value = 0.0003837 Shapiro-Wilk: W = 0.98934, p-value = 0.9945 Levene's Test (center = median): F(2, 21) = 3.4123, Pr(>F) = 0.0521 $method diff lwr upr p adj Practice-Reread 10.5 3.785985 17.214015 0.0020668 Flashcard-Reread 8.5 1.785985 15.214015 0.0117201 Flashcard-Practice -2.0 -8.714015 4.714015 0.7364417 ``` Games-Howell is not in base R or in car; packages such as rstatix and PMCMRplus provide it. This short function implements it from the formula with base R's studentised range distribution (`ptukey`, `qtukey`). Run it in the same R session as the block above (it uses `d`): ```r games_howell <- function(y, g) { lv <- levels(g); out <- NULL for (i in 1:(length(lv) - 1)) for (j in (i + 1):length(lv)) { a <- y[g == lv[i]]; b <- y[g == lv[j]] va <- var(a) / length(a); vb <- var(b) / length(b) diff <- mean(b) - mean(a); se <- sqrt(va + vb) df <- (va + vb)^2 / (va^2 / (length(a) - 1) + vb^2 / (length(b) - 1)) p <- ptukey(abs(diff) / se * sqrt(2), length(lv), df, lower.tail = FALSE) crit <- qtukey(0.95, length(lv), df) / sqrt(2) out <- rbind(out, data.frame(pair = paste(lv[j], "-", lv[i]), diff = diff, lwr = round(diff - crit * se, 2), upr = round(diff + crit * se, 2), df = round(df, 2), p = round(p, 4))) } out } games_howell(d$score, d$method) ``` ```text pair diff lwr upr df p 1 Practice - Reread 10.5 5.29 15.71 11.62 0.0005 2 Flashcard - Reread 8.5 0.67 16.33 9.07 0.0345 3 Flashcard - Practice -2.0 -10.28 6.28 11.84 0.7985 ``` Effect sizes, the 95% CI for η² (by inverting the noncentral F distribution, Steiger 2004) and the planned contrast. This block continues the R session above (it reuses `d` and `fit`): ```r tab <- summary(fit)[[1]] ssb <- tab[1, "Sum Sq"]; ssw <- tab[2, "Sum Sq"]; msw <- tab[2, "Mean Sq"] eta2 <- ssb / (ssb + ssw); omega2 <- (ssb - 2 * msw) / (ssb + ssw + msw) Fobs <- tab[1, "F value"]; N <- 24 ncp_lo <- uniroot(function(n) pf(Fobs, 2, 21, n) - 0.975, c(0, 200))$root ncp_hi <- uniroot(function(n) pf(Fobs, 2, 21, n) - 0.025, c(0, 200))$root c(eta2 = eta2, omega2 = omega2, lo = ncp_lo / (ncp_lo + N), hi = ncp_hi / (ncp_hi + N)) m <- tapply(d$score, d$method, mean); w <- c(-1, 0.5, 0.5) L <- sum(w * m); se <- sqrt(msw * sum(w^2 / 8)) c(L = L, SE = se, t = L / se, p = 2 * pt(-abs(L / se), 21)) ``` ```text eta2 omega2 lo hi 0.45487805 0.39276617 0.09719016 0.62976107 L SE t p 9.5000000000 2.3068221803 4.1182194628 0.0004897752 ``` **Python code and output.** ```python import numpy as np import pandas as pd from scipy import stats from scipy.optimize import brentq import statsmodels.formula.api as smf from statsmodels.stats.anova import anova_lm from statsmodels.stats.oneway import anova_oneway from statsmodels.stats.multicomp import pairwise_tukeyhsd d = pd.DataFrame({ "method": ["Reread"] * 8 + ["Practice"] * 8 + ["Flashcard"] * 8, "score": [58, 62, 55, 64, 60, 57, 61, 59, 70, 66, 74, 63, 78, 69, 72, 68, 65, 72, 58, 80, 61, 75, 69, 64]}) names = ["Reread", "Practice", "Flashcard"] groups = [d.score[d.method == m].to_numpy() for m in names] fit = smf.ols("score ~ C(method)", data=d).fit() print(anova_lm(fit)) # classic ANOVA welch = anova_oneway(d.score, d.method, use_var="unequal") # Welch ANOVA print(welch.statistic, welch.df_denom, welch.pvalue) print(stats.levene(*groups, center="median"), stats.shapiro(fit.resid)) print(pairwise_tukeyhsd(d.score, d.method)) # Tukey HSD gh = stats.tukey_hsd(*groups, equal_var=False) # Games-Howell (recent SciPy; run on 1.18.1) ci = gh.confidence_interval() for i, j in [(1, 0), (2, 0), (2, 1)]: print(names[i], "-", names[j], groups[i].mean() - groups[j].mean(), round(ci.low[i, j], 2), round(ci.high[i, j], 2), round(gh.pvalue[i, j], 4)) def eta2_ci(F, df1, df2, level=0.95): a = (1 - level) / 2 cdf = lambda ncp: stats.ncf.cdf(F, df1, df2, ncp) lo = 0.0 if stats.f.cdf(F, df1, df2) < 1 - a else brentq(lambda n: cdf(n) - (1 - a), 1e-9, 500) hi = brentq(lambda n: cdf(n) - a, 1e-9, 500) N = df1 + df2 + 1 return round(lo / (lo + N), 3), round(hi / (hi + N), 3) tab = anova_lm(fit) ssb, ssw, msw = tab.sum_sq.iloc[0], tab.sum_sq.iloc[1], tab.mean_sq.iloc[1] print(round(ssb / (ssb + ssw), 3), round((ssb - 2 * msw) / (ssb + ssw + msw), 3), eta2_ci(tab.F.iloc[0], 2, 21)) w = np.array([-1, 0.5, 0.5]); m = np.array([g.mean() for g in groups]) L = w @ m; se = np.sqrt(msw * np.sum(w**2 / 8)) print(L, round(se, 3), round(L / se, 3), round(2 * stats.t.sf(abs(L / se), 21), 4)) ``` ```text df sum_sq mean_sq F PR(>F) C(method) 2.0 497.333333 248.666667 8.761745 0.001711 Residual 21.0 596.000000 28.380952 NaN NaN 15.796387430609785 12.45234664226742 0.0003837347334856708 LeveneResult(statistic=np.float64(3.4122562674094707), pvalue=np.float64(0.0520970541855913)) ShapiroResult(statistic=np.float64(0.9893375295174673), pvalue=np.float64(0.9944708622838384)) Multiple Comparison of Means - Tukey HSD, FWER=0.05 ======================================================== group1 group2 meandiff p-adj lower upper reject -------------------------------------------------------- Flashcard Practice 2.0 0.7364 -4.714 8.714 False Flashcard Reread -8.5 0.0117 -15.214 -1.786 True Practice Reread -10.5 0.0021 -17.214 -3.786 True -------------------------------------------------------- Practice - Reread 10.5 5.29 15.71 0.0005 Flashcard - Reread 8.5 0.67 16.33 0.0345 Flashcard - Practice -2.0 -10.28 6.28 0.7985 0.455 0.393 (0.097, 0.63) 9.5 2.307 4.118 0.0005 ``` R and Python agree on every number. Two presentation differences matter: statsmodels' `pairwise_tukeyhsd` sorts the groups alphabetically and reports group2 minus group1 (so "Flashcard, Reread, −8.5" means Reread is 8.5 lower), while R's `TukeyHSD` follows the factor level order. And R's `oneway.test` is Welch unless you set `var.equal = TRUE`, whereas `scipy.stats.f_oneway` and statsmodels' `anova_lm` give the classic F; for Welch in Python, ask for `use_var="unequal"` explicitly. **By hand.** Grand mean = (59.5 + 70 + 68) / 3 = 65.833 (equal n). Between groups: $SS_B = 8[(59.5-65.833)^2 + (70-65.833)^2 + (68-65.833)^2] = 8(40.11 + 17.36 + 4.69) = 497.33$. Within groups: $SS_W = 7(2.878^2 + 4.690^2 + 7.407^2) = 7(8.29 + 22.00 + 54.86) = 596.0$. Then $MS_B = 497.33/2 = 248.67$, $MS_W = 596/21 = 28.38$, and $F = 248.67/28.38 = 8.76$ with df (2, 21). **Effect size with CI.** η² = 497.33 / 1093.33 = 0.455, 95% CI [0.097, 0.630]; ω² = (497.33 − 2 x 28.38) divided by (1093.33 + 28.38) = 0.393. The CI is wide because n = 8 per group: the data are consistent with anything from a small to a very large share of variance explained. **95% or 90%?** The code above gives a two-sided 95% interval (2.5% in each tail of the noncentral F). Steiger (2004) shows that a **90%** interval matches the F test for η², and several effect-size texts recommend it, because η² cannot be negative and the F test is one-sided: a 90% interval excludes 0 exactly when the F test is significant at α = 0.05, so the interval and the test agree. With `level = 0.90` the same function gives [0.143, 0.600] here. Either is defensible; state the level you used, and do not compare a 90% interval from one paper with a 95% interval from another. Report ω² (less biased) as the main effect size, or the mean differences with their CIs, which are easier to interpret. **Interpretation.** Revision method made a difference to quiz scores. Welch's ANOVA, which is appropriate given the unequal SDs, gives F(2, 12.45) = 15.80, p < .001. The planned contrast shows that the two active methods together beat rereading by 9.5 points (t(21) = 4.12, p < .001). Games-Howell comparisons show that Practice beat Reread by 10.5 points (95% CI [5.29, 15.71]) and Flashcard beat Reread by 8.5 points (95% CI [0.67, 16.33]), while Practice and Flashcard did not differ detectably (−2.0, 95% CI [−10.28, 6.28]). That last interval is wide: it does not show the two active methods are equal. Welch's F is larger than the classic F here because the low-variance Reread group, which differs most from the others, gets more weight. **APA-style report.** Quiz scores differed by revision method, Welch's *F*(2, 12.45) = 15.80, *p* < .001, ω² = .39 (classic ANOVA: *F*(2, 21) = 8.76, *p* = .002, η² = .46, 95% CI [.10, .63]). Students who used practice tests (*M* = 70.0, *SD* = 4.7) or flashcards (*M* = 68.0, *SD* = 7.4) scored higher than those who reread notes (*M* = 59.5, *SD* = 2.9); Games-Howell: Practice − Reread = 10.5, 95% CI [5.3, 15.7], *p* < .001; Flashcard − Reread = 8.5, 95% CI [0.7, 16.3], *p* = .035; Practice − Flashcard = 2.0, 95% CI [−6.3, 10.3], *p* = .799. ## Worked example B (half solved): Brightwell Bakery ovens **Question.** Brightwell Bakery (fictional) baked six loaves from the same dough batch in each of three ovens and measured loaf volume (cm³). Do the ovens differ in mean volume? | Oven | Volumes (cm³) | Mean | SD | |---|---|---|---| | *Example:* A | 1510, 1495, 1520, 1505, 1500, 1515 | 1507.5 | 9.35 | | *Example:* B | 1530, 1525, 1540, 1520, 1535, 1530 | 1530.0 | 7.07 | | *Example:* C | 1500, 1490, 1505, 1495, 1510, 1500 | 1500.0 | 7.07 | ```r b <- data.frame(oven = factor(rep(c("A", "B", "C"), each = 6)), volume = c(1510, 1495, 1520, 1505, 1500, 1515, 1530, 1525, 1540, 1520, 1535, 1530, 1500, 1490, 1505, 1495, 1510, 1500)) fb <- aov(volume ~ oven, b); summary(fb); oneway.test(volume ~ oven, b); TukeyHSD(fb) ``` ```python import pandas as pd from scipy import stats from statsmodels.stats.oneway import anova_oneway from statsmodels.stats.multicomp import pairwise_tukeyhsd b = pd.DataFrame({"oven": ["A"] * 6 + ["B"] * 6 + ["C"] * 6, "volume": [1510, 1495, 1520, 1505, 1500, 1515, 1530, 1525, 1540, 1520, 1535, 1530, 1500, 1490, 1505, 1495, 1510, 1500]}) print(stats.f_oneway(*[b.volume[b.oven == o] for o in "ABC"])) print(anova_oneway(b.volume, b.oven, use_var="unequal")) print(pairwise_tukeyhsd(b.volume, b.oven)) ``` ```text (output abridged: R printouts condensed) Df Sum Sq Mean Sq F value Pr(>F) oven 2 2925.0 1462.5 23.4 2.44e-05 *** Residuals 15 937.5 62.5 Welch: F = 26.696, num df = 2.000, denom df = 9.863, p-value = 0.0001047 diff lwr upr p adj B-A 22.5 10.64422 34.355775 0.0005018 C-A -7.5 -19.35578 4.355775 0.2588712 C-B -30.0 -41.85578 -18.144225 0.0000250 ``` Python gives the same values (`f_oneway`: statistic 23.4, p = 2.44e-05; Welch statistic 26.696, df_denom 9.863). 1. Assumption check: SD ratio = 9.35 / 7.07 = [ ]. Is the classic ANOVA reasonable here? [ ] 2. Grand mean = [ ]. $SS_B = 6[(1507.5 - \bar y)^2 + (1530 - \bar y)^2 + (1500 - \bar y)^2]$ = [ ] (check: 2925). 3. $SS_W = 5(9.35^2 + 7.07^2 + 7.07^2)$ = [ ]. $MS_W$ = [ ] with df = [ ]. 4. F = [ ] with df ([ ], [ ]); p [ ]. 5. η² = [ ]; ω² = [ ]. 6. Which pairs differ (Tukey, α = 0.05)? [ ] 7. Report sentence: [ ] ## Exercise C (unsolved): Harbour City Library reshelving times Harbour City Library (fictional) timed how long it took to reshelve a trolley of 50 books (minutes) at three branches, five trolleys per branch. Only the summaries were recorded: | Branch | n | Mean (min) | SD (min) | |---|---|---|---| | *Example:* Central | 5 | 12.0 | 2.0 | | *Example:* Harbourside | 5 | 15.0 | 3.0 | | *Example:* Northgate | 5 | 18.0 | 2.5 | Tasks: 1. State H0 and H1. 2. Compute $SS_B$, $SS_W$, the mean squares and F. Fill a complete ANOVA table. 3. Find the p-value (software: `pf(F, 2, 12, lower.tail = FALSE)` in R or `stats.f.sf(F, 2, 12)` in Python). 4. Compute η² and ω². 5. Which follow-up procedure would you use, and why? Write the APA sentence. ## Reading software output | Output line | What it means | What to report | |---|---|---| | `method Df 2 Sum Sq 497.3 Mean Sq 248.67 F 8.762 Pr(>F) 0.00171` | Between-group row: SS_B, df = k − 1, MS_B, F and its p | *F*(2, 21) = 8.76, *p* = .002 | | `Residuals Df 21 Sum Sq 596.0 Mean Sq 28.38` | Within-group row: SS_W, df = N − k, MS_W (pooled variance) | The second df; MS_W for hand calculations | | `F = 15.796, num df = 2.000, denom df = 12.452` | Welch's ANOVA; the denominator df is estimated and usually not an integer | *F*(2, 12.45) = 15.80 | | `Levene's Test ... F value 3.4123 Pr(>F) 0.0521` | Test of equal variances (median-centred) | Mention, but justify the Welch choice by SDs and design | | `TukeyHSD: diff, lwr, upr, p adj` | Pairwise difference, family-wise 95% CI, adjusted p | Difference with CI and adjusted p for each pair | | `pairwise_tukeyhsd: meandiff` | group2 − group1, groups sorted alphabetically | Re-sign to match your write-up | | `reject True/False` (statsmodels) | Whether adjusted p < 0.05 | Do not report; give the p-value and CI | | Games-Howell `df` | Welch-type df for each pair | Not usually reported; the CI and p are | ## Reporting template **Fill-in sentence.** [Outcome] [differed / did not differ detectably] across [factor] groups, [Welch's / classic] *F*([df1], [df2]) = [value], *p* = [.xxx], ω² = [.xx] [95% CI for η² [lo, hi]]. [Follow-up procedure] showed that [group A] (*M* = [ ], *SD* = [ ]) scored [higher / lower] than [group B] (*M* = [ ], *SD* = [ ]), difference = [ ], 95% CI [[lo], [hi]], *p* = [.xxx]. | Group | n | Mean | SD | 95% CI for mean | |---|---|---|---|---| | *Example:* Reread | 8 | 59.5 | 2.9 | [57.1, 61.9] | | *Example:* Practice | 8 | 70.0 | 4.7 | [66.1, 73.9] | | *Example:* Flashcard | 8 | 68.0 | 7.4 | [61.8, 74.2] | | [Group] | [ ] | [ ] | [ ] | [ ] | Results checklist: - [ ] Design, group sizes and how units were assigned - [ ] Descriptives per group (n, mean, SD) in a table or figure with CIs - [ ] Assumption checks reported briefly, and the reason for classic or Welch - [ ] Omnibus F with both df and p in APA style - [ ] Effect size: ω² (or η² with CI), and mean differences with CIs - [ ] Planned contrasts or a named post hoc procedure with adjusted p-values - [ ] Non-significant pairs described as "not detectably different", with their CIs - [ ] Causal language only if units were randomly assigned ## Common mistakes and misconceptions 1. **Running all pairwise t-tests without adjustment.** The family-wise false-positive rate grows with the number of pairs; use Tukey, Games-Howell or planned contrasts. 2. **Stopping at a significant F.** F says "some means differ", not which. Report the follow-up comparisons and effect sizes. 3. **Choosing classic or Welch from Levene's p-value.** A non-significant Levene test with small groups is not evidence of equal variances (Example A: p = .052 with an SD ratio of 2.6). 4. **"A non-significant pair means the methods are equally good."** Practice versus Flashcard has a 95% CI from about −6 to 10 points: the data cannot rule out a meaningful difference. 5. **Checking normality of the raw outcome pooled across groups.** Check the residuals (or each group); a mixture of shifted normal groups is not normal. 6. **Reporting η² as if it were unbiased.** In small samples η² overstates the population effect; ω² is smaller (0.393 vs 0.455 here). 7. **Post hoc tests chosen after looking at which one gives significance.** Choose the procedure in your analysis plan. 8. **Using one-way ANOVA for repeated measures.** The same students in every condition violate independence and waste the pairing. 9. **Forgetting that software defaults differ.** R's `oneway.test` is Welch by default; SciPy's `f_oneway` is classic; state which you used. 10. **"F = 8.76 means the active methods are 8.76 times better."** F is a ratio of variance estimates, not of means. ## Self-test questions 1. A study has 4 groups of 10. What are the df for the classic F test? 2. Explain in two sentences why F is close to 1 when H0 is true. 3. Group SDs are 3, 4 and 9, with n = 6 per group. Which omnibus test and which post hoc procedure do you choose? 4. SS_B = 120, SS_W = 380, k = 3, N = 33. Compute F, η² and ω². 5. Why can the ANOVA be significant while no Tukey pair is significant? 6. What is the difference between a planned contrast and a post hoc comparison, and why does it matter for error control? ## My notes and next steps | Item | My entry | |---|---| | Confidence (1-5) in building the ANOVA table by hand | [ ] | | Confidence (1-5) in choosing classic vs Welch and the follow-up test | [ ] | | What I still find confusing | [ ] | | Questions for my tutor or supervisor | [ ] | | Mistakes logged in academic/stats-error-log | [Link or entry IDs] | | Next review date (spacing) | YYYY-MM-DD | | Next template to study | [academic/stats-factorial-anova or academic/stats-repeated-measures] | ## Sources and further reading Structure aligned with the method-card shape of this statistics library; methods based on the original papers and the public-domain NIST/SEMATECH e-Handbook, written in original words with fictional examples. - Fisher, R. A. (1925). *Statistical Methods for Research Workers*. Oliver and Boyd. - Welch, B. L. (1951). On the comparison of several mean values: an alternative approach. *Biometrika*, 38(3–4), 330–336. - Tukey, J. W. (1949). Comparing individual means in the analysis of variance. *Biometrics*, 5(2), 99–114. - Tukey, J. W. (1953). *The Problem of Multiple Comparisons*. Unpublished manuscript. - Kramer, C. Y. (1956). Extension of multiple range tests to group means with unequal numbers of replications. *Biometrics*, 12(3), 307–310. - Games, P. A., and Howell, J. F. (1976). Pairwise multiple comparison procedures with unequal n's and/or variances: a Monte Carlo study. *Journal of Educational Statistics*, 1(2), 113–125. - Brown, M. B., and Forsythe, A. B. (1974). Robust tests for the equality of variances. *Journal of the American Statistical Association*, 69(346), 364–367. - Hays, W. L. (1963). *Statistics for Psychologists*. Holt, Rinehart and Winston (the usual cited source of ω²). - Cohen, J. (1988). *Statistical Power Analysis for the Behavioral Sciences* (2nd ed.). Lawrence Erlbaum Associates. - Steiger, J. H. (2004). Beyond the F test: effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. *Psychological Methods*, 9(2), 164–182. - NIST/SEMATECH e-Handbook of Statistical Methods, sections 7.4.3 Are the means equal? and 7.4.7 How can we make multiple comparisons?, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain. - R Core Team. Documentation for `aov`, `oneway.test`, `TukeyHSD`, `ptukey` (R 4.5.1); Fox and Weisberg, car package documentation for `leveneTest` (3.1-5). - SciPy documentation for `f_oneway`, `levene`, `tukey_hsd` (1.18.1); statsmodels documentation for `anova_lm`, `anova_oneway`, `pairwise_tukeyhsd` (0.15.0). - Further reading (by name only): de Smith, M. J. (2024). *Statistical Analysis Handbook*, 2024 edition. StatsRef, https://www.statsref.com/; OpenIntro Statistics; APA *Publication Manual* (7th ed.) and APA JARS-Quant. ## Answer key **Worked example B.** 1. SD ratio = 1.32; with equal n and similar SDs, the classic ANOVA with Tukey HSD is reasonable (Welch agrees: F(2, 9.86) = 26.70, p < .001). 2. Grand mean = 1512.5; $SS_B = 6(25 + 306.25 + 156.25) = 6 \times 487.5 = 2925$. 3. $SS_W = 5(87.5 + 50 + 50) = 937.5$; $MS_W = 937.5 / 15 = 62.5$ with df = 15. 4. $MS_B = 2925 / 2 = 1462.5$; F = 1462.5 / 62.5 = 23.4 with df (2, 15); p < .001 (2.44 x 10⁻⁵). 5. η² = 2925 / 3862.5 = 0.757; ω² = (2925 − 125) / (3862.5 + 62.5) = 0.713. 6. B differs from A (22.5 cm³, 95% CI [10.6, 34.4], p < .001) and from C (30.0 cm³, 95% CI [18.1, 41.9], p < .001); A and C do not differ detectably (7.5 cm³, 95% CI [−4.4, 19.4], p = .259). 7. Loaf volume differed by oven, *F*(2, 15) = 23.40, *p* < .001, ω² = .71; Tukey HSD showed that oven B produced larger loaves than ovens A and C, which did not differ detectably. **Exercise C.** H0: the three branch means are equal. Grand mean = 15.0. $SS_B = 5(9 + 0 + 9) = 90$, df = 2, $MS_B = 45$. $SS_W = 4(4 + 9 + 6.25) = 77$, df = 12, $MS_W = 6.417$. F = 45 / 6.417 = 7.01, *p* = .010 (0.0096). η² = 90 / 167 = 0.539; ω² = (90 − 12.83) / (167 + 6.42) = 0.445. With equal n and SDs from 2.0 to 3.0, Tukey HSD is a reasonable follow-up (Games-Howell if you prefer not to assume equal variances; with only the summaries, you can compute either by hand from the means, SDs and n). Sentence: Reshelving time differed by branch, *F*(2, 12) = 7.01, *p* = .010, ω² = .45. **Self-test.** 1. df = (k − 1, N − k) = (3, 36). 2. Under H0 both the between-group mean square and the within-group mean square estimate the same error variance, so their ratio fluctuates around 1. Real differences in means add to the between-group mean square only. 3. The SD ratio is 3 with small equal groups, so Welch's ANOVA and Games-Howell comparisons. 4. $MS_B = 60$, $MS_W = 380/30 = 12.67$, F(2, 30) = 4.74; η² = 120 / 500 = 0.24; ω² = (120 − 25.33) / (500 + 12.67) = 0.185. 5. The omnibus F can detect a pattern spread over several means (for example a linear trend, or one mean against the average of the others) that no single pair captures after the multiple-comparison adjustment. Planned contrasts test such patterns directly. 6. A planned contrast is specified before seeing the data and tests one focused hypothesis, so it uses its own α (adjust only if there are several). Post hoc comparisons look at all pairs after seeing the data, so they need a family-wise adjustment such as Tukey or Games-Howell. ## Revision history | Version | Date | Author | Change | |---|---|---|---| | 0.1 | YYYY-MM-DD | [Your name] | First version from the template | | [0.2] | YYYY-MM-DD | [ ] | [Replaced example data with my own] |