Statistical Test Selector Template (Statistics, Decision Trees, R and Python)

Statistical Test Selector: [Your research question or course]

A decision tool for choosing the right statistical test or model. Three decision trees (comparing groups, relationships and prediction, categorical data and counts) and four master tables map your outcome type, predictor type, number of groups, design (paired or independent) and assumptions to the parametric choice, the nonparametric or robust alternative, the effect size to report, the R and Python functions, and the method card that teaches it. It is for students who freeze at "which test do I use?", for master's and PhD students planning a thesis analysis, and for analysts who want a quick, defensible choice. It speeds up learning by turning test choice into five questions asked in a fixed order, by showing one selection fully worked, one half worked and one for you, and by closing with self-test scenarios whose answers sit in a separate key. The worked examples use fictional data; replace them with your own. Source basis: the NIST/SEMATECH e-Handbook of Statistical Methods, the original method papers (Student 1908, Fisher 1925, Welch 1947, Mann and Whitney 1947, Nelder and Wedderburn 1972) and the R and SciPy/statsmodels documentation, written up in original words and examples.

Document control

How to use the selector

A statistical test is a tool built for one kind of question asked of one kind of data. Choosing it is not a matter of memorising a long list of names: almost every introductory and intermediate method can be reached by answering five questions about your study, in a fixed order. The order matters, because the early questions (what is the outcome, what is the aim) eliminate whole families of methods, and the later ones (pairing, assumptions) choose within a family.

Work in this order:

1. Write down your research question in one sentence, naming the outcome and the predictor or grouping variable.
2. Answer the five questions in the next section. Write the answers down; do not keep them in your head.
3. Follow the decision tree that matches your aim. Each leaf names a method and its fallback.
4. Find the row in the master tables. It gives the parametric choice, the alternative, the effect size, the R and Python functions and the method card slug.
5. Check the assumptions with academic/stats-assumption-checklist before you run the test, and decide in advance what you will do if a check fails. Deciding after you see the p-value is a form of p-hacking.
6. Record the choice and the reason in your analysis plan (academic/stats-analysis-plan).

Structure aligned with the decision logic common to introductory and intermediate courses (outcome type, design, assumptions) and with the NIST/SEMATECH e-Handbook's grouping of methods; the tables, wording and examples are original.

Two principles run through the whole selector. First, the design decides, not the data: whether observations are paired, clustered or independent follows from how the data were collected, and no test on the data can tell you. Second, prefer estimation to a bare test: every row names an effect size, because a p-value alone says nothing about how large or important an effect is (see the ASA statement on p-values, Wasserstein and Lazar 2016).

Five questions to ask before choosing

A sixth question that is easy to forget: is the analysis confirmatory or exploratory? A confirmatory test was planned before seeing the data, with its alpha (α = 0.05 unless stated), sidedness (two-sided by default) and correction for multiple tests fixed in advance. An exploratory analysis can try several models, but its p-values must be labelled as exploratory.

Measurement levels in brief. Nominal: categories with no order (faculty, bakery product). Ordinal: ordered categories with unequal or unknown spacing (a single Likert item, a ranking, a pain scale). Interval: equal spacing, no true zero (temperature in Celsius). Ratio: equal spacing and a true zero (time, weight, exam points). A count is a non-negative whole number of events (library loans per day). A binary variable has two values (passed or not). Sum scores of many Likert items are usually treated as numeric; a single item is ordinal.

The decision trees

The three trees below cover the methods in this library. Start at the top of the tree that matches your aim, answer the diamonds, and stop at a leaf. The leaf gives a default and, where it matters, a fallback for when the assumptions fail. Default choices follow the library canon: Welch's t-test for two independent groups, Welch ANOVA when group spreads differ, and a model (regression, GLM or mixed model) whenever covariates or clustering are involved.

Tree 1: comparing groups on a numeric or ordinal outcome

Use this tree when the outcome is a number or an ordered rating and the question is whether groups or conditions differ. The first split is the number of groups; the second is whether the same units appear in more than one group.

Tree 2: relationships and prediction

Use this tree when you want to measure how strongly two variables move together, or to predict an outcome from one or more predictors. The type of the outcome (numeric, binary, ordinal, count, time series) chooses the model.

Tree 3: categorical data and counts

Use this tree when the data are categories or proportions: one categorical variable against expected shares, two categorical variables against each other, or the same units classified twice.

Read the dotted arrows as "consider this as well": once you have a baseline covariate (a pre-test score, age), the group comparison is usually better done as ANCOVA or regression, because adjusting for a strong covariate reduces residual variance and raises power.

Master table 1: comparing groups

Use this table when the aim is to compare groups or conditions on a numeric or ordinal outcome. All tests are two-sided at α = 0.05 unless your plan states otherwise.

Master table 2: relationships and prediction

Master table 3: categorical outcomes and counts

Master table 4: cross-cutting tools

Where R and Python defaults differ, the difference changes results, so check these before comparing software output:

• Independent t-test: R t.test defaults to Welch (var.equal = FALSE); SciPy ttest_ind defaults to the pooled test (equal_var=True). Pass equal_var=False in Python.
• ANOVA sums of squares: R anova() and aov() use sequential (type I) sums of squares; statsmodels anova_lm uses type I by default too, but many tutorials pass typ=2. With unbalanced factorial data the choice changes the F tests (see academic/stats-factorial-anova).
• 2 x 2 chi-square: both R chisq.test and SciPy chi2_contingency apply the Yates continuity correction by default for a 2 x 2 table; switch it off with correct = FALSE or correction=False if your course does not use it.
• Fisher's exact test for tables larger than 2 x 2: R computes the exact p-value; SciPy 1.18 estimates it by Monte Carlo simulation (two runs without a seed give slightly different p-values), so pass a seeded MonteCarloMethod to make the result reproducible.
• McNemar test: R mcnemar.test gives the chi-square version with a continuity correction; statsmodels mcnemar gives the exact binomial version by default (exact=True). Pass exact=False in Python to match R.
• Rank tests with ties: R wilcox.test switches from the exact to the normal approximation when there are ties and warns; SciPy mannwhitneyu chooses its method automatically. Small p-value differences are expected.

Why question 3 matters: one data set, two designs

Ten Lakemoor University (fictional) students took a statistics quiz before and after a two-hour revision workshop. Each student has two scores, so the design is paired. The code runs the correct paired t-test and, for contrast, the Welch t-test that treats the two columns as if they came from twenty different students.

before <- c(54, 61, 48, 70, 66, 58, 75, 52, 63, 59)
after  <- c(58, 63, 53, 71, 70, 60, 79, 55, 68, 61)
t.test(after, before, paired = TRUE)
t.test(after, before)              # Welch: WRONG design for these data
diff <- after - before
cat("mean diff =", mean(diff), " sd diff =", round(sd(diff), 3), " dz =", round(mean(diff)/sd(diff), 2), "\n")

(output abridged)
	Paired t-test
t = 7.2363, df = 9, p-value = 4.888e-05
95 percent confidence interval:
 2.199636 4.200364
mean difference 
            3.2 

	Welch Two Sample t-test
t = 0.87417, df = 17.991, p-value = 0.3935
95 percent confidence interval:
 -4.49091 10.89091

mean diff = 3.2  sd diff = 1.398  dz = 2.29

import numpy as np
from scipy import stats

before = np.array([54, 61, 48, 70, 66, 58, 75, 52, 63, 59])
after = np.array([58, 63, 53, 71, 70, 60, 79, 55, 68, 61])
print(stats.ttest_rel(after, before))
print(stats.ttest_ind(after, before, equal_var=False))   # WRONG design
d = after - before
print("mean diff =", d.mean(), " sd diff =", round(d.std(ddof=1), 3), " dz =", round(d.mean() / d.std(ddof=1), 2))

TtestResult(statistic=np.float64(7.236272269866326), pvalue=np.float64(4.8879962427790686e-05), df=np.int64(9))
TtestResult(statistic=np.float64(0.8741733835330437), pvalue=np.float64(0.3935436647168462), df=np.float64(17.99111549851925))
mean diff = 3.2  sd diff = 1.398  dz = 2.29

The mean gain is 3.2 points in both analyses. The paired test compares that gain with the spread of the differences (SD 1.40), which is small because every student improved by a similar amount: t(9) = 7.24, p < .001, 95% CI [2.20, 4.20]. The Welch test compares it with the spread between students (scores range from 48 to 79), which is large: t(18.0) = 0.87, p = .394, 95% CI [−4.49, 10.89]. Same numbers, opposite conclusions. The chart shows why: every line slopes upwards, but the lines start at very different heights.

Worked selection example A (fully solved)

Scenario. Brightwell Bakery (fictional) tests three proofing times (40, 60 and 80 minutes) for its sourdough. Each of 30 loaves is randomly assigned to one proofing time, 10 per time, and the loaf volume (in millilitres) is measured. The question: does mean loaf volume differ by proofing time?

Step 1: the five questions.

Step 2: follow Tree 1. Three or more groups, not the same units, one factor, so the leaf is one-way ANOVA (Welch ANOVA if spreads differ), with Kruskal-Wallis as the fallback.

Step 3: confirm in master table 1. Row "Numeric, 3+ independent groups, one factor". Effect size: ω² (less biased than η² in small samples). Post hoc: Tukey HSD if spreads are similar, Games-Howell if not. R: oneway.test(volume ~ time) and aov(); Python: anova_oneway(..., use_var="unequal"). Method card: academic/stats-one-way-anova.

Step 4: plan the checks (from academic/stats-assumption-checklist). Residual QQ plot by eye; group SDs (ratio of largest to smallest under about 2 suggests the classic F is fine); a dot plot per group to spot outliers.

Step 5: think about the alternative. Proofing time is a numeric factor with three ordered levels. If the question were "how much does volume change per extra 10 minutes?", linear regression with time as a numeric predictor (or a polynomial contrast) would answer it more directly. Write this as a secondary analysis in the plan.

Step 6: write the decision. "Loaf volume will be compared across the three proofing times with Welch's one-way ANOVA, followed by Games-Howell pairwise comparisons; ω² with 95% CI will be reported. If residuals are strongly non-normal, a Kruskal-Wallis test with Holm-adjusted Dunn tests will be reported instead."

Worked selection example B (half solved)

Scenario. Harbour City Library (fictional) asked 120 members whether they would use a weekend opening (yes or no), then repeated the same question to the same 120 members after a trial weekend opening.

Step 2: Tree 3. Same units classified twice, two occasions, so the leaf is [ ].

Step 3: master table 3. Effect size: [ ]. R function: mcnemar.test(tab). Python function: [ ].

Step 4: the trap. A chi-square test of independence on the 2 x 2 table of "time by answer" would be wrong because [ ].

Exercise C (unsolved)

Scenario. Lakemoor University's sports centre (fictional) records, for 80 students, the number of gym visits in the autumn term (a count from 0 to 40), the student's year of study (1 to 4) and whether the student lives on campus (yes or no). The question: do on-campus students visit more often, after allowing for year of study?

Write: (a) the answers to the five questions; (b) the tree and leaf you reach; (c) the check you would run first on the fitted model, and the method you would switch to if that check fails; (d) the effect size you would report.

Common mistakes and misconceptions

Self-test scenarios

1. Thirty Lakemoor University students rate a lecture on a single 1 to 5 item, and twenty-eight students from another section rate a different lecturer on the same item. Which test compares the two sections?
2. Brightwell Bakery records the number of customer complaints per week for 52 weeks, along with whether a new packaging line was running that week. Which model, and what must you check about the counts?
3. A researcher measures reaction time for 24 participants under three noise levels, each participant doing all three. The data are complete and roughly normal. Which test, and which extra assumption does it bring?
4. True or false: "If my data fail a normality test, I must use a nonparametric test." Explain in one sentence.
5. Harbour City Library compares the proportion of overdue loans in two branches (412 loans and 389 loans). Which test, and which effect size?
6. Why is a 2 x 2 chi-square test on 15 students in total risky, and what do you use instead?

My notes and next steps

Next step: practise on thirty mixed scenarios in academic/stats-practice-which-test, then open the method card for the method your own study needs.

Sources and further reading

• NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.5 Quantitative Techniques and chapter 7 Product and Process Comparisons, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain. Followed for method grouping and assumption checks.
• Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25.
• Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd.
• Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1–2), 28–35.
• Mann, H. B., and Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18(1), 50–60.
• McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153–157.
• Nelder, J. A., and Wedderburn, R. W. M. (1972). Generalized linear models. Journal of the Royal Statistical Society, Series A, 135(3), 370–384.
• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum.
• Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133.
• American Statistical Association (2016). Guidelines for Assessment and Instruction in Statistics Education (GAISE) College Report.
• Further reading (by name only): de Smith, M. J., Statistical Analysis Handbook (2024 edition); Diez, Çetinkaya-Rundel and Barr, OpenIntro Statistics.
• R Core Team, R: A Language and Environment for Statistical Computing (stats package documentation); SciPy and statsmodels documentation.

Answer key

Example B.

• Question 3: paired. The same 120 members answered both times.
• Tree 3 leaf: McNemar test (same units classified twice, two occasions).
• Effect size: the paired odds ratio (ratio of the two discordant counts: members who switched from no to yes divided by members who switched from yes to no), or the difference in proportions with a CI.
• Python: statsmodels.stats.contingency_tables.mcnemar(tab), where tab is the 2 x 2 table of before (rows) by after (columns).
• The trap: a chi-square test of independence would treat the 240 answers as 240 independent people, but there are only 120 people, each counted twice. McNemar uses only the members who changed their answer, which is where the information about change lies.

Exercise C.

• (a) Outcome: gym visits, a count. Aim: compare on-campus and off-campus students, adjusted for year, so it is a prediction model. Design: independent students (one record each). Predictors: one binary (on campus) and one categorical or numeric (year). Assumptions: count model assumptions, especially variance relative to the mean.
• (b) Tree 2: there is an outcome, observations are independent, the outcome is a count, so Poisson regression with visits ~ on_campus + year; if the variance is much larger than the mean, negative binomial regression.
• (c) First check: overdispersion, the Pearson chi-square divided by the residual df (values well above 1, say 1.5 or more, signal overdispersion). Gym visits usually vary far more than a Poisson model allows (many students never go, a few go every day), so expect to switch to negative binomial (MASS::glm.nb() in R, smf.negativebinomial() in Python). With many zeros, mention a zero-inflated model as a sensitivity analysis.
• (d) Effect size: the rate ratio for on campus vs off campus, exp(coefficient), with its 95% CI, for example "on-campus students made 1.4 times as many visits (95% CI [lower, upper])". Method card: academic/stats-count-regression.

Self-test scenarios.

1. Mann-Whitney U test (single Likert item is ordinal; two independent sections). Effect size: rank-biserial r. Check: independence of raters, and similar distribution shapes if you want to describe the result as a median difference.
2. Poisson regression of weekly complaints on the packaging-line indicator, effect size the rate ratio. Check overdispersion (switch to negative binomial if present) and autocorrelation between consecutive weeks (plot residuals over time).
3. Repeated-measures ANOVA (same participants, three conditions, complete data). It adds the sphericity assumption; check with Mauchly's test and report the Greenhouse-Geisser correction. Effect size: partial or generalised η². A linear mixed model is the alternative if data were missing.
4. False. Normality is about residuals (or the sampling distribution of the mean), tests are oversensitive with large n and underpowered with small n, and the t-test is robust to moderate non-normality when n is moderate; look at plots and think about the variable before switching.
5. Two-proportion z-test (or chi-square on the 2 x 2 table, which is equivalent without continuity correction), with the risk difference and its Newcombe CI, plus the risk ratio or odds ratio. Check that each branch has at least about 10 overdue and 10 not overdue loans, and that loans are independent (one borrower with many loans breaks this).
6. With 15 students the expected counts in some cells will almost certainly be below 5, so the chi-square approximation is unreliable. Use Fisher's exact test and report the odds ratio with its exact CI.

Revision history