# Statistical Test Selector: [Your research question or course] > A decision tool for choosing the right statistical test or model. Three decision trees (comparing groups, > relationships and prediction, categorical data and counts) and four master tables map your outcome type, predictor > type, number of groups, design (paired or independent) and assumptions to the parametric choice, the nonparametric or > robust alternative, the effect size to report, the R and Python functions, and the method card that teaches it. It is > for students who freeze at "which test do I use?", for master's and PhD students planning a thesis analysis, and for > analysts who want a quick, defensible choice. It speeds up learning by turning test choice into five questions asked in > a fixed order, by showing one selection fully worked, one half worked and one for you, and by closing with self-test > scenarios whose answers sit in a separate key. The worked examples use fictional data; replace them with your own. > Source basis: the NIST/SEMATECH e-Handbook of Statistical Methods, the original method papers (Student 1908, Fisher > 1925, Welch 1947, Mann and Whitney 1947, Nelder and Wedderburn 1972) and the R and SciPy/statsmodels documentation, > written up in original words and examples. ## Document control | Field | Value | |---|---| | Course or project | [STAT 201 / thesis chapter 4 / client project] | | Learner or analyst | [Your name] | | Data set | [Lakemoor University study-habits survey (fictional) / your data set and source] | | Software and versions | R 4.5.1 (stats); Python 3.14 with numpy 2.5.4, scipy 1.18.1, pandas 3.1.0, statsmodels 0.15.0 | | Linked analysis plan | [academic/stats-analysis-plan, version 0.x] | | Date | YYYY-MM-DD | | Version | [0.1] | ## How to use the selector A statistical test is a tool built for one kind of question asked of one kind of data. Choosing it is not a matter of memorising a long list of names: almost every introductory and intermediate method can be reached by answering five questions about your study, in a fixed order. The order matters, because the early questions (what is the outcome, what is the aim) eliminate whole families of methods, and the later ones (pairing, assumptions) choose within a family. Work in this order: 1. Write down your research question in one sentence, naming the outcome and the predictor or grouping variable. 2. Answer the five questions in the next section. Write the answers down; do not keep them in your head. 3. Follow the decision tree that matches your aim. Each leaf names a method and its fallback. 4. Find the row in the master tables. It gives the parametric choice, the alternative, the effect size, the R and Python functions and the method card slug. 5. Check the assumptions with `academic/stats-assumption-checklist` before you run the test, and decide in advance what you will do if a check fails. Deciding after you see the p-value is a form of p-hacking. 6. Record the choice and the reason in your analysis plan (`academic/stats-analysis-plan`). Structure aligned with the decision logic common to introductory and intermediate courses (outcome type, design, assumptions) and with the NIST/SEMATECH e-Handbook's grouping of methods; the tables, wording and examples are original. Two principles run through the whole selector. First, **the design decides, not the data**: whether observations are paired, clustered or independent follows from how the data were collected, and no test on the data can tell you. Second, **prefer estimation to a bare test**: every row names an effect size, because a p-value alone says nothing about how large or important an effect is (see the ASA statement on p-values, Wasserstein and Lazar 2016). ## Five questions to ask before choosing | # | Question | Why it matters | Typical answers | My study | |---|---|---|---|---| | 1 | What is the **outcome** and what is its measurement level? | The outcome type picks the family: means, ranks, proportions, counts, times | Numeric (continuous), ordinal (ranks, Likert item), binary, nominal with 3+ categories, count, time series | [ ] | | 2 | What is the **aim**: compare groups, measure a relationship, predict, or describe agreement? | Comparing groups leads to t-tests and ANOVA; relationships to correlation and regression | Compare / relate / predict / agree | [ ] | | 3 | Are observations **paired, repeated or clustered**, or fully independent? | Ignoring pairing throws away power; ignoring clustering gives false precision | Independent / same people twice / same people 3+ times / students within classes | [ ] | | 4 | **How many groups** or predictors, and of what type? | Two groups vs three or more changes the test; covariates move you to a model | 1, 2, 3+ groups; one or several predictors; numeric or categorical | [ ] | | 5 | Are the **assumptions** plausible for this design and sample size? | Decides between the parametric choice and the alternative | Approximate normality of residuals, equal spread, expected counts at least 5, linearity, no overdispersion | [ ] | A sixth question that is easy to forget: **is the analysis confirmatory or exploratory?** A confirmatory test was planned before seeing the data, with its alpha (α = 0.05 unless stated), sidedness (two-sided by default) and correction for multiple tests fixed in advance. An exploratory analysis can try several models, but its p-values must be labelled as exploratory. Measurement levels in brief. **Nominal**: categories with no order (faculty, bakery product). **Ordinal**: ordered categories with unequal or unknown spacing (a single Likert item, a ranking, a pain scale). **Interval**: equal spacing, no true zero (temperature in Celsius). **Ratio**: equal spacing and a true zero (time, weight, exam points). A **count** is a non-negative whole number of events (library loans per day). A **binary** variable has two values (passed or not). Sum scores of many Likert items are usually treated as numeric; a single item is ordinal. ## The decision trees The three trees below cover the methods in this library. Start at the top of the tree that matches your aim, answer the diamonds, and stop at a leaf. The leaf gives a default and, where it matters, a fallback for when the assumptions fail. Default choices follow the library canon: Welch's t-test for two independent groups, Welch ANOVA when group spreads differ, and a model (regression, GLM or mixed model) whenever covariates or clustering are involved. ### Tree 1: comparing groups on a numeric or ordinal outcome Use this tree when the outcome is a number or an ordered rating and the question is whether groups or conditions differ. The first split is the number of groups; the second is whether the same units appear in more than one group. ```mermaid flowchart TD A{"How many groups or conditions?"} -->|"One, against a fixed value"| B{"Roughly normal or n of 30+?"} B -->|Yes| B1["One-sample t-test"] B -->|"No, or ordinal"| B2["Wilcoxon signed-rank vs the value"] A -->|Two| C{"Same units in both, or matched pairs?"} C -->|"Yes, paired"| D{"Differences roughly normal?"} D -->|Yes| D1["Paired t-test"] D -->|No| D2["Wilcoxon signed-rank test"] C -->|"No, independent"| E{"Each group roughly normal or n of 30+?"} E -->|Yes| E1["Welch t-test"] E -->|"No, or ordinal"| E2["Mann-Whitney U test"] A -->|"Three or more"| F{"Same units in every condition?"} F -->|No| G{"One factor or two or more?"} G -->|One| H{"Residuals roughly normal?"} H -->|Yes| H1["One-way ANOVA, Welch if spreads differ"] H -->|"No, or ordinal"| H2["Kruskal-Wallis with Dunn tests"] G -->|"Two or more"| I["Factorial ANOVA"] F -->|Yes| J{"Outcome type and completeness of the data?"} J -->|"Numeric, complete, same occasions"| J1["Repeated-measures ANOVA with sphericity check"] J -->|"Ordinal, or strongly non-normal"| J2["Friedman test"] J -->|"Missing data or unequal timing"| J3["Linear mixed model"] E1 -.->|"Baseline covariate to adjust?"| K["ANCOVA or regression"] H1 -.->|"Baseline covariate to adjust?"| K ``` ### Tree 2: relationships and prediction Use this tree when you want to measure how strongly two variables move together, or to predict an outcome from one or more predictors. The type of the outcome (numeric, binary, ordinal, count, time series) chooses the model. ```mermaid flowchart TD A{"Is there an outcome to predict?"} -->|"No, association of two variables"| B{"Both numeric, linear, no strong outliers?"} B -->|Yes| B1["Pearson correlation"] B -->|"No, monotonic or ordinal"| B2["Spearman or Kendall correlation"] A -->|Yes| C{"Observations independent?"} C -->|"No, clustered or repeated"| C1["Mixed model: lmer or glmer"] C -->|Yes| D{"What type is the outcome?"} D -->|Numeric| E{"Main interest is a group effect adjusted for a covariate?"} E -->|Yes| E1["ANCOVA"] E -->|No| E2["Linear regression"] D -->|Binary| F1["Logistic regression"] D -->|Ordinal| F2["Ordinal logistic regression"] D -->|Count| G{"Variance close to the mean?"} G -->|Yes| G1["Poisson regression"] G -->|"No, overdispersed"| G2["Negative binomial regression"] D -->|"One series over time"| H1["ARIMA time series model"] A -->|"Several items measure one construct"| I1["Reliability and factor analysis"] ``` ### Tree 3: categorical data and counts Use this tree when the data are categories or proportions: one categorical variable against expected shares, two categorical variables against each other, or the same units classified twice. ```mermaid flowchart TD A{"What are you analysing?"} -->|"One categorical variable vs expected shares"| B{"Two categories?"} B -->|Yes| B1["Exact binomial test or one-proportion z with Wilson CI"] B -->|"Three or more"| B2["Chi-square goodness of fit"] A -->|"Two categorical variables, independent units"| C{"Expected counts large enough? See the rule in master table 3"} C -->|Yes| C1["Chi-square test of independence and Cramer's V"] C -->|No| C2["Fisher's exact test"] C1 -.->|"2 x 2 table"| C3["Two-proportion z, risk difference, OR and RR"] A -->|"Same units classified twice"| D{"How many occasions?"} D -->|Two| D1["McNemar test"] D -->|"Three or more"| D2["Cochran's Q or mixed logistic model"] A -->|"Binary outcome with several predictors"| E1["Logistic regression"] A -->|"Counts per unit of time or space"| F1["Poisson or negative binomial regression"] ``` Read the dotted arrows as "consider this as well": once you have a baseline covariate (a pre-test score, age), the group comparison is usually better done as ANCOVA or regression, because adjusting for a strong covariate reduces residual variance and raises power. ## Master table 1: comparing groups Use this table when the aim is to compare groups or conditions on a numeric or ordinal outcome. All tests are two-sided at α = 0.05 unless your plan states otherwise. | Outcome | Groups and design | Key assumptions | Parametric choice | If assumptions fail | Effect size | R | Python | Method card | |---|---|---|---|---|---|---|---|---| | Numeric | 1 group vs a fixed value | Independence; approx. normal or n of 30+ | One-sample t-test | Wilcoxon signed-rank vs value; bootstrap CI | Cohen's d | `t.test(x, mu = m)` | `scipy.stats.ttest_1samp(x, m)` | `academic/stats-t-tests` | | Numeric | 2 independent groups | Independence; approx. normal per group or n of 30+ each | Welch t-test | Mann-Whitney U; bootstrap CI of mean difference | Hedges' g (or d), CI of difference | `t.test(y ~ g)` | `scipy.stats.ttest_ind(a, b, equal_var=False)` | `academic/stats-t-tests` | | Ordinal or skewed numeric | 2 independent groups | Independence; similar shapes if you want to talk about medians | Mann-Whitney U (Wilcoxon rank-sum) | Permutation test | Rank-biserial r, Hodges-Lehmann shift | `wilcox.test(y ~ g)` | `scipy.stats.mannwhitneyu(a, b)` | `academic/stats-nonparametric-tests` | | Numeric | 2 paired measures (same units) | Differences approx. normal | Paired t-test | Wilcoxon signed-rank; sign test | d_z (mean difference / SD of differences) | `t.test(a, b, paired = TRUE)` | `scipy.stats.ttest_rel(a, b)` | `academic/stats-t-tests` | | Ordinal or skewed numeric | 2 paired measures | Differences symmetric (signed-rank) | Wilcoxon signed-rank | Sign test (no symmetry needed) | Matched rank-biserial r | `wilcox.test(a, b, paired = TRUE)` | `scipy.stats.wilcoxon(a, b)` | `academic/stats-nonparametric-tests` | | Numeric | 3+ independent groups, one factor | Normal residuals; equal variances for classic F | One-way ANOVA (Welch ANOVA if spreads differ), Tukey HSD or Games-Howell | Kruskal-Wallis with Dunn tests | η², ω² | `oneway.test(y ~ g)`, `aov()`, `TukeyHSD()` | `statsmodels.stats.oneway.anova_oneway(..., use_var="unequal")`, `pairwise_tukeyhsd` | `academic/stats-one-way-anova` | | Ordinal or skewed numeric | 3+ independent groups | Independence | Kruskal-Wallis | Permutation ANOVA | ε² (epsilon squared) or η²_H | `kruskal.test(y ~ g)` | `scipy.stats.kruskal(*groups)` | `academic/stats-nonparametric-tests` | | Numeric | 2+ crossed factors, independent | Normal residuals, equal variances, balanced or near | Factorial ANOVA with interaction | Robust or permutation approach; transform | Partial η² | `fit <- lm(y ~ A * B)`, `car::Anova(fit, type = 2)` (type 3 only with sum-to-zero contrasts) | `fit = smf.ols("y ~ C(A) * C(B)", data).fit()`, `sm.stats.anova_lm(fit, typ=2)` | `academic/stats-factorial-anova` | | Numeric | Same units in 3+ conditions | Normal residuals; sphericity | Repeated-measures ANOVA with Greenhouse-Geisser | Friedman test; linear mixed model | Partial or generalised η² | `aov(y ~ cond + Error(id / cond))` (uncorrected); Mauchly and Greenhouse-Geisser from `anova(lm(cbind(c1, c2, c3) ~ 1), X = ~1, test = "Spherical")` | `statsmodels.stats.anova.AnovaRM` (uncorrected F only; see `academic/stats-assumption-checklist` for Greenhouse-Geisser) | `academic/stats-repeated-measures` | | Ordinal | Same units in 3+ conditions | Complete blocks | Friedman test | Mixed ordinal model | Kendall's W | `friedman.test(y ~ cond \| id)` | `scipy.stats.friedmanchisquare(c1, c2, c3)` | `academic/stats-nonparametric-tests` | | Numeric | Groups plus a numeric covariate | Linearity, homogeneity of slopes | ANCOVA | Regression with interaction; robust SE | Partial η², adjusted mean difference | `lm(y ~ x + g)`, `emmeans::emmeans()` | `smf.ols("y ~ x + C(g)")` | `academic/stats-ancova` | | Numeric | Repeated or clustered, missing data allowed | Normal residuals and random effects | Linear mixed model | GEE; cluster-robust SE | Fixed-effect estimates with CI, ICC | `lme4::lmer(y ~ t + (1 \| id))` | `smf.mixedlm("y ~ t", data, groups=data["id"])` | `academic/stats-mixed-models` | ## Master table 2: relationships and prediction | Outcome | Predictors | Key assumptions | Parametric choice | If assumptions fail | Effect size | R | Python | Method card | |---|---|---|---|---|---|---|---|---| | None (two numeric variables) | One numeric | Linear relation, no extreme outliers, bivariate approx. normal for the CI | Pearson r | Spearman ρ; Kendall τ for small n or many ties | r (with CI), r² | `cor.test(x, y)` | `scipy.stats.pearsonr(x, y)` | `academic/stats-correlation` | | None (ordinal or monotonic) | One ordinal or numeric | Monotonic relation | Spearman ρ | Kendall τ | ρ or τ | `cor.test(x, y, method = "spearman")` | `scipy.stats.spearmanr(x, y)`, `scipy.stats.kendalltau(x, y)` | `academic/stats-correlation` | | Numeric | One or more numeric or categorical | Linearity, independence, constant variance, normal residuals | Linear regression (OLS) | Transform; robust (HC3) SE; bootstrap | R², adjusted R², standardised β | `lm(y ~ x1 + x2)` | `smf.ols("y ~ x1 + x2", data).fit()` | `academic/stats-linear-regression` | | Binary | One or more | Independence, linear in the logit, no separation, enough events per parameter | Logistic regression | Penalised (Firth) logistic; exact methods | Odds ratio with CI, marginal effects | `glm(y ~ x, family = binomial)` | `smf.logit("y ~ x", data).fit()` | `academic/stats-logistic-regression` | | Ordinal (3+ ordered levels) | One or more | Proportional odds | Ordinal (proportional-odds) logistic regression | Multinomial logistic | Cumulative odds ratio | `MASS::polr(y ~ x, Hess = TRUE)` | `statsmodels.miscmodels.ordinal_model.OrderedModel.from_formula("y ~ x", data, distr="logit")` | `academic/stats-logistic-regression` (binary card; ordinal models only mentioned there, so use the functions named here and their documentation) | | Count | One or more | Mean equals variance; log-linear | Poisson regression (with offset for exposure) | Negative binomial; quasi-Poisson | Rate ratio with CI | `glm(y ~ x, family = poisson)`, `MASS::glm.nb()` | `smf.glm("y ~ x", data, family=sm.families.Poisson())`, `smf.negativebinomial()` | `academic/stats-count-regression` | | Numeric, binary or count, clustered | Fixed effects plus grouping | Correct random-effects structure | Mixed model (LMM or GLMM) | GEE; cluster-robust SE | Fixed effects with CI, ICC | `lme4::lmer()`, `lme4::glmer()` | `smf.mixedlm()` | `academic/stats-mixed-models` | | One series over time | Its own past | Stationarity after differencing | ARIMA | Exponential smoothing; seasonal naive benchmark | Forecast accuracy (MAE, RMSE, MASE) | `arima()`, `forecast::auto.arima()` | `statsmodels.tsa.arima.model.ARIMA` | `academic/stats-time-series-arima` | | Scale of several items | Items | Items measure one construct | Cronbach's α, exploratory factor analysis | McDonald's ω; PCA for data reduction only | α, loadings, variance explained | `factanal()`, `prcomp()` | `statsmodels.multivariate.factor.Factor` | `academic/stats-reliability-factor-analysis` | ## Master table 3: categorical outcomes and counts | Data | Design | Key assumptions | Parametric or large-sample choice | Small-sample or paired alternative | Effect size | R | Python | Method card | |---|---|---|---|---|---|---|---|---| | One binary variable vs a reference proportion | Independent trials | n·p and n·(1 − p) at least about 10 for z | One-proportion z-test, Wilson CI | Exact binomial test | Difference from reference, Cohen's h | `prop.test(x, n, p = p0)`, `binom.test(x, n, p0)` | `statsmodels.stats.proportion.proportions_ztest`, `scipy.stats.binomtest(x, n, p0)` | `academic/stats-proportion-tests` | | One nominal variable, 3+ categories | Independent units | Expected count 5+ in every cell | Chi-square goodness of fit | Exact multinomial or Monte Carlo p | Cohen's w | `chisq.test(counts, p = shares)` | `scipy.stats.chisquare(counts, f_exp)` | `academic/stats-contingency-tables` | | Two binary variables (2 x 2) | Independent units | Expected 5+ per cell | Chi-square or two-proportion z | Fisher's exact test | Risk difference, RR, OR, φ | `chisq.test(tab)`, `prop.test(x, n)`, `fisher.test(tab)` | `scipy.stats.chi2_contingency(tab)`, `scipy.stats.fisher_exact(tab)` | `academic/stats-contingency-tables` | | Two nominal variables (r x c) | Independent units | Expected 5+ in at least 80% of cells, none below 1 | Chi-square test of independence | Fisher's exact (R exact, SciPy Monte Carlo for r x c) | Cramér's V | `chisq.test(tab)`, `fisher.test(tab)` | `scipy.stats.chi2_contingency(tab)`, `scipy.stats.fisher_exact(tab, method=scipy.stats.MonteCarloMethod(rng=...))` | `academic/stats-contingency-tables` | | Binary measured twice on the same units | Paired | Enough discordant pairs (about 10+) for the chi-square version | McNemar test | Exact McNemar (binomial on discordant pairs) | Paired OR (ratio of discordant counts) | `mcnemar.test(tab)` (chi-square with continuity correction) | `statsmodels.stats.contingency_tables.mcnemar(tab)` (exact binomial by default; `exact=False` for chi-square) | `academic/stats-contingency-tables` | | Binary measured 3+ times on the same units | Paired | Complete blocks | Cochran's Q | Mixed logistic model | Differences in proportions | `lme4::glmer(y ~ t + (1 \| id), family = binomial)` (base R has no Cochran's Q) | `statsmodels.stats.contingency_tables.cochrans_q(x)` (x: units in rows, occasions in columns, 0 or 1) | `academic/stats-contingency-tables` (McNemar background; Cochran's Q is only named in this selector), `academic/stats-mixed-models` (`glmer`) | | Two proportions, independent groups | Independent | Large-sample: 10+ successes and failures per group | Two-proportion z-test, Newcombe CI | Fisher's exact | Risk difference, RR, OR | `prop.test(c(x1, x2), c(n1, n2))` | `statsmodels.stats.proportion.test_proportions_2indep` | `academic/stats-proportion-tests` | ## Master table 4: cross-cutting tools | Need | Tool | When it helps | R | Python | Template | |---|---|---|---|---|---| | Assumptions doubtful, sample small | Bootstrap CI | Any statistic, especially medians, ratios, differences | `boot::boot()`, `boot::boot.ci()` | `scipy.stats.bootstrap` | `academic/stats-resampling-methods` | | Exact p-value without distributional assumptions | Permutation test | Two groups, correlation, small n | Loop with `sample()` | `scipy.stats.permutation_test` | `academic/stats-resampling-methods` | | Many tests at once | Multiplicity correction | Several outcomes, many pairwise comparisons | `p.adjust(p, "holm")`, `p.adjust(p, "BH")` | `statsmodels.stats.multitest.multipletests` | `academic/stats-hypothesis-testing` | | Planning the sample size | Power analysis | Before data collection | `power.t.test()`, `pwr::pwr.anova.test()` | `statsmodels.stats.power.TTestIndPower` | `academic/stats-power-sample-size` | | Skewed numeric data | Transformation | Right-skew, multiplicative effects | `log()`, `MASS::boxcox()` | `numpy.log`, `scipy.stats.boxcox` | `academic/stats-data-transformations` | | Showing "no meaningful difference" | Equivalence test (TOST) | A non-significant result is not evidence of no effect | Two one-sided `t.test()` calls | `statsmodels.stats.weightstats.ttost_ind` | `academic/stats-hypothesis-testing` | Where R and Python defaults differ, the difference changes results, so check these before comparing software output: - **Independent t-test**: R `t.test` defaults to Welch (`var.equal = FALSE`); SciPy `ttest_ind` defaults to the pooled test (`equal_var=True`). Pass `equal_var=False` in Python. - **ANOVA sums of squares**: R `anova()` and `aov()` use sequential (type I) sums of squares; statsmodels `anova_lm` uses type I by default too, but many tutorials pass `typ=2`. With unbalanced factorial data the choice changes the F tests (see `academic/stats-factorial-anova`). - **2 x 2 chi-square**: both R `chisq.test` and SciPy `chi2_contingency` apply the Yates continuity correction by default for a 2 x 2 table; switch it off with `correct = FALSE` or `correction=False` if your course does not use it. - **Fisher's exact test for tables larger than 2 x 2**: R computes the exact p-value; SciPy 1.18 estimates it by Monte Carlo simulation (two runs without a seed give slightly different p-values), so pass a seeded `MonteCarloMethod` to make the result reproducible. - **McNemar test**: R `mcnemar.test` gives the chi-square version with a continuity correction; statsmodels `mcnemar` gives the exact binomial version by default (`exact=True`). Pass `exact=False` in Python to match R. - **Rank tests with ties**: R `wilcox.test` switches from the exact to the normal approximation when there are ties and warns; SciPy `mannwhitneyu` chooses its method automatically. Small p-value differences are expected. ## Why question 3 matters: one data set, two designs Ten Lakemoor University (fictional) students took a statistics quiz before and after a two-hour revision workshop. Each student has two scores, so the design is **paired**. The code runs the correct paired t-test and, for contrast, the Welch t-test that treats the two columns as if they came from twenty different students. ```r before <- c(54, 61, 48, 70, 66, 58, 75, 52, 63, 59) after <- c(58, 63, 53, 71, 70, 60, 79, 55, 68, 61) t.test(after, before, paired = TRUE) t.test(after, before) # Welch: WRONG design for these data diff <- after - before cat("mean diff =", mean(diff), " sd diff =", round(sd(diff), 3), " dz =", round(mean(diff)/sd(diff), 2), "\n") ``` ```text (output abridged) Paired t-test t = 7.2363, df = 9, p-value = 4.888e-05 95 percent confidence interval: 2.199636 4.200364 mean difference 3.2 Welch Two Sample t-test t = 0.87417, df = 17.991, p-value = 0.3935 95 percent confidence interval: -4.49091 10.89091 mean diff = 3.2 sd diff = 1.398 dz = 2.29 ``` ```python import numpy as np from scipy import stats before = np.array([54, 61, 48, 70, 66, 58, 75, 52, 63, 59]) after = np.array([58, 63, 53, 71, 70, 60, 79, 55, 68, 61]) print(stats.ttest_rel(after, before)) print(stats.ttest_ind(after, before, equal_var=False)) # WRONG design d = after - before print("mean diff =", d.mean(), " sd diff =", round(d.std(ddof=1), 3), " dz =", round(d.mean() / d.std(ddof=1), 2)) ``` ```text TtestResult(statistic=np.float64(7.236272269866326), pvalue=np.float64(4.8879962427790686e-05), df=np.int64(9)) TtestResult(statistic=np.float64(0.8741733835330437), pvalue=np.float64(0.3935436647168462), df=np.float64(17.99111549851925)) mean diff = 3.2 sd diff = 1.398 dz = 2.29 ``` The mean gain is 3.2 points in both analyses. The paired test compares that gain with the spread of the **differences** (SD 1.40), which is small because every student improved by a similar amount: *t*(9) = 7.24, *p* < .001, 95% CI [2.20, 4.20]. The Welch test compares it with the spread **between students** (scores range from 48 to 79), which is large: *t*(18.0) = 0.87, *p* = .394, 95% CI [−4.49, 10.89]. Same numbers, opposite conclusions. The chart shows why: every line slopes upwards, but the lines start at very different heights. ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "description": "Quiz score before and after the workshop for ten students, one line per student", "width": 320, "height": 240, "data": {"values": [ {"student": "S01", "time": "1 Before", "score": 54}, {"student": "S01", "time": "2 After", "score": 58}, {"student": "S02", "time": "1 Before", "score": 61}, {"student": "S02", "time": "2 After", "score": 63}, {"student": "S03", "time": "1 Before", "score": 48}, {"student": "S03", "time": "2 After", "score": 53}, {"student": "S04", "time": "1 Before", "score": 70}, {"student": "S04", "time": "2 After", "score": 71}, {"student": "S05", "time": "1 Before", "score": 66}, {"student": "S05", "time": "2 After", "score": 70}, {"student": "S06", "time": "1 Before", "score": 58}, {"student": "S06", "time": "2 After", "score": 60}, {"student": "S07", "time": "1 Before", "score": 75}, {"student": "S07", "time": "2 After", "score": 79}, {"student": "S08", "time": "1 Before", "score": 52}, {"student": "S08", "time": "2 After", "score": 55}, {"student": "S09", "time": "1 Before", "score": 63}, {"student": "S09", "time": "2 After", "score": 68}, {"student": "S10", "time": "1 Before", "score": 59}, {"student": "S10", "time": "2 After", "score": 61} ]}, "mark": {"type": "line", "point": true}, "encoding": { "x": {"field": "time", "type": "ordinal", "title": "Occasion"}, "y": {"field": "score", "type": "quantitative", "title": "Quiz score", "scale": {"zero": false}}, "detail": {"field": "student"}, "color": {"value": "#4c78a8"} } } ``` ## Worked selection example A (fully solved) **Scenario.** Brightwell Bakery (fictional) tests three proofing times (40, 60 and 80 minutes) for its sourdough. Each of 30 loaves is randomly assigned to one proofing time, 10 per time, and the loaf volume (in millilitres) is measured. The question: does mean loaf volume differ by proofing time? **Step 1: the five questions.** | # | Question | Answer | |---|---|---| | 1 | Outcome and level | Loaf volume, numeric (ratio scale) | | 2 | Aim | Compare groups | | 3 | Paired or independent | Independent: each loaf gets one proofing time | | 4 | Number of groups | Three groups of one factor (proofing time) | | 5 | Assumptions plausible? | Volume is continuous and roughly symmetric in past batches; check residuals and spreads | **Step 2: follow Tree 1.** Three or more groups, not the same units, one factor, so the leaf is **one-way ANOVA (Welch ANOVA if spreads differ)**, with Kruskal-Wallis as the fallback. **Step 3: confirm in master table 1.** Row "Numeric, 3+ independent groups, one factor". Effect size: ω² (less biased than η² in small samples). Post hoc: Tukey HSD if spreads are similar, Games-Howell if not. R: `oneway.test(volume ~ time)` and `aov()`; Python: `anova_oneway(..., use_var="unequal")`. Method card: `academic/stats-one-way-anova`. **Step 4: plan the checks** (from `academic/stats-assumption-checklist`). Residual QQ plot by eye; group SDs (ratio of largest to smallest under about 2 suggests the classic F is fine); a dot plot per group to spot outliers. **Step 5: think about the alternative.** Proofing time is a numeric factor with three ordered levels. If the question were "how much does volume change per extra 10 minutes?", linear regression with time as a numeric predictor (or a polynomial contrast) would answer it more directly. Write this as a secondary analysis in the plan. **Step 6: write the decision.** "Loaf volume will be compared across the three proofing times with Welch's one-way ANOVA, followed by Games-Howell pairwise comparisons; ω² with 95% CI will be reported. If residuals are strongly non-normal, a Kruskal-Wallis test with Holm-adjusted Dunn tests will be reported instead." ## Worked selection example B (half solved) **Scenario.** Harbour City Library (fictional) asked 120 members whether they would use a weekend opening (yes or no), then repeated the same question to the same 120 members after a trial weekend opening. | # | Question | Answer | |---|---|---| | 1 | Outcome and level | Would use weekend opening, binary (yes or no) | | 2 | Aim | Compare the proportion saying yes before and after | | 3 | Paired or independent | [ ] | | 4 | Number of occasions | Two | | 5 | Assumptions plausible? | Need enough members who changed their answer (discordant pairs) | **Step 2: Tree 3.** Same units classified twice, two occasions, so the leaf is [ ]. **Step 3: master table 3.** Effect size: [ ]. R function: `mcnemar.test(tab)`. Python function: [ ]. **Step 4: the trap.** A chi-square test of independence on the 2 x 2 table of "time by answer" would be wrong because [ ]. ## Exercise C (unsolved) **Scenario.** Lakemoor University's sports centre (fictional) records, for 80 students, the number of gym visits in the autumn term (a count from 0 to 40), the student's year of study (1 to 4) and whether the student lives on campus (yes or no). The question: do on-campus students visit more often, after allowing for year of study? Write: (a) the answers to the five questions; (b) the tree and leaf you reach; (c) the check you would run first on the fitted model, and the method you would switch to if that check fails; (d) the effect size you would report. ## Common mistakes and misconceptions | Mistake | Why it is wrong | Do this instead | |---|---|---| | Choosing the test after seeing which one gives *p* < .05 | Inflates the false-positive rate well above α | Fix the method, the fallback and the decision rule in the analysis plan first | | Analysing paired data with an independent-groups test | Ignores that each unit is its own control; loses power, as the demonstration above shows | Decide pairing from the design: same units, matched pairs, or twins means paired | | Running many t-tests instead of one ANOVA | With three groups and three pairwise tests at α = 0.05, the chance of at least one false positive rises to about 14% | One-way ANOVA (or Welch ANOVA) then a post hoc procedure that controls the family-wise error | | Using a Shapiro-Wilk p-value as a switch between t-test and Mann-Whitney | With large n the test flags harmless departures; with small n it misses serious ones | Look at plots, think about the variable, and use robust defaults (Welch, bootstrap) | | Thinking nonparametric tests have no assumptions | Rank tests still need independence, and Mann-Whitney only compares medians when the two shapes are similar | State what the rank test actually compares (the probability that a random member of one group exceeds one of the other) | | Using chi-square when expected counts are small | The chi-square approximation breaks down; p-values can be badly off | Check the expected table; use Fisher's exact test or merge sparse categories with a reason | | Using a chi-square test of independence on paired yes or no data | Counts the same people twice and treats them as independent | McNemar test for two occasions, Cochran's Q or a mixed logistic model for more | | Treating a single 5-point Likert item as normal and continuous without thought | A single item is ordinal with few levels | Use rank tests or ordinal regression; sum scales of many items can usually be treated as numeric | | Running Pearson correlation on a curved or outlier-driven pattern | r measures linear association only and is sensitive to outliers | Plot first; use Spearman for monotonic patterns and report the influential point | | Reading "not significant" as "no difference" | Absence of evidence is not evidence of absence, especially with small n | Report the CI; use an equivalence test (TOST) if you want to claim no meaningful difference | | Ignoring clustering (students within classes, loaves within batches) | Standard errors are too small, so p-values are too small | Mixed model or cluster-robust SE (`academic/stats-mixed-models`) | ## Self-test scenarios 1. Thirty Lakemoor University students rate a lecture on a single 1 to 5 item, and twenty-eight students from another section rate a different lecturer on the same item. Which test compares the two sections? 2. Brightwell Bakery records the number of customer complaints per week for 52 weeks, along with whether a new packaging line was running that week. Which model, and what must you check about the counts? 3. A researcher measures reaction time for 24 participants under three noise levels, each participant doing all three. The data are complete and roughly normal. Which test, and which extra assumption does it bring? 4. True or false: "If my data fail a normality test, I must use a nonparametric test." Explain in one sentence. 5. Harbour City Library compares the proportion of overdue loans in two branches (412 loans and 389 loans). Which test, and which effect size? 6. Why is a 2 x 2 chi-square test on 15 students in total risky, and what do you use instead? ## My notes and next steps | Item | My entry | |---|---| | Confidence choosing a test (1 to 5) | [ ] | | The question I find hardest of the five | [ ] | | Confusable pairs I mix up | [e.g. McNemar vs chi-square] | | Open questions for my supervisor or tutor | [ ] | | Mistakes logged in `academic/stats-error-log` | [link or count] | | Next review date (spacing) | YYYY-MM-DD | Next step: practise on thirty mixed scenarios in `academic/stats-practice-which-test`, then open the method card for the method your own study needs. ## Sources and further reading - NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.5 Quantitative Techniques and chapter 7 Product and Process Comparisons, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain. Followed for method grouping and assumption checks. - Student (1908). The probable error of a mean. *Biometrika*, 6(1), 1–25. - Fisher, R. A. (1925). *Statistical Methods for Research Workers*. Oliver and Boyd. - Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. *Biometrika*, 34(1–2), 28–35. - Mann, H. B., and Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. *Annals of Mathematical Statistics*, 18(1), 50–60. - McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. *Psychometrika*, 12(2), 153–157. - Nelder, J. A., and Wedderburn, R. W. M. (1972). Generalized linear models. *Journal of the Royal Statistical Society, Series A*, 135(3), 370–384. - Cohen, J. (1988). *Statistical Power Analysis for the Behavioral Sciences* (2nd ed.). Lawrence Erlbaum. - Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. *The American Statistician*, 70(2), 129–133. - American Statistical Association (2016). *Guidelines for Assessment and Instruction in Statistics Education (GAISE) College Report*. - Further reading (by name only): de Smith, M. J., *Statistical Analysis Handbook* (2024 edition); Diez, Çetinkaya-Rundel and Barr, *OpenIntro Statistics*. - R Core Team, *R: A Language and Environment for Statistical Computing* (stats package documentation); SciPy and statsmodels documentation. ## Answer key **Example B.** - Question 3: **paired**. The same 120 members answered both times. - Tree 3 leaf: **McNemar test** (same units classified twice, two occasions). - Effect size: the **paired odds ratio** (ratio of the two discordant counts: members who switched from no to yes divided by members who switched from yes to no), or the difference in proportions with a CI. - Python: `statsmodels.stats.contingency_tables.mcnemar(tab)`, where `tab` is the 2 x 2 table of before (rows) by after (columns). - The trap: a chi-square test of independence would treat the 240 answers as 240 independent people, but there are only 120 people, each counted twice. McNemar uses only the members who changed their answer, which is where the information about change lies. **Exercise C.** - (a) Outcome: gym visits, a **count**. Aim: compare on-campus and off-campus students, adjusted for year, so it is a prediction model. Design: independent students (one record each). Predictors: one binary (on campus) and one categorical or numeric (year). Assumptions: count model assumptions, especially variance relative to the mean. - (b) Tree 2: there is an outcome, observations are independent, the outcome is a count, so **Poisson regression** with `visits ~ on_campus + year`; if the variance is much larger than the mean, **negative binomial regression**. - (c) First check: **overdispersion**, the Pearson chi-square divided by the residual df (values well above 1, say 1.5 or more, signal overdispersion). Gym visits usually vary far more than a Poisson model allows (many students never go, a few go every day), so expect to switch to negative binomial (`MASS::glm.nb()` in R, `smf.negativebinomial()` in Python). With many zeros, mention a zero-inflated model as a sensitivity analysis. - (d) Effect size: the **rate ratio** for on campus vs off campus, exp(coefficient), with its 95% CI, for example "on-campus students made 1.4 times as many visits (95% CI [lower, upper])". Method card: `academic/stats-count-regression`. **Self-test scenarios.** 1. **Mann-Whitney U test** (single Likert item is ordinal; two independent sections). Effect size: rank-biserial r. Check: independence of raters, and similar distribution shapes if you want to describe the result as a median difference. 2. **Poisson regression** of weekly complaints on the packaging-line indicator, effect size the rate ratio. Check **overdispersion** (switch to negative binomial if present) and **autocorrelation** between consecutive weeks (plot residuals over time). 3. **Repeated-measures ANOVA** (same participants, three conditions, complete data). It adds the **sphericity** assumption; check with Mauchly's test and report the Greenhouse-Geisser correction. Effect size: partial or generalised η². A linear mixed model is the alternative if data were missing. 4. **False.** Normality is about residuals (or the sampling distribution of the mean), tests are oversensitive with large n and underpowered with small n, and the t-test is robust to moderate non-normality when n is moderate; look at plots and think about the variable before switching. 5. **Two-proportion z-test** (or chi-square on the 2 x 2 table, which is equivalent without continuity correction), with the **risk difference** and its Newcombe CI, plus the risk ratio or odds ratio. Check that each branch has at least about 10 overdue and 10 not overdue loans, and that loans are independent (one borrower with many loans breaks this). 6. With 15 students the **expected counts** in some cells will almost certainly be below 5, so the chi-square approximation is unreliable. Use **Fisher's exact test** and report the odds ratio with its exact CI. ## Revision history | Version | Date | Author | Change summary | |---|---|---|---| | 0.1 | YYYY-MM-DD | [Your name] | First draft from the template | | 0.2 | YYYY-MM-DD | [Your name] | *Example:* Five questions answered for thesis study 2; Welch ANOVA chosen |