# t-Tests Method Card: [Your study question or course] > A method card for the three t-tests you will meet in every statistics course and most thesis projects: the > one-sample t-test, the independent two-sample t-test (Welch by default, pooled only when justified) and the paired > t-test. It is for students preparing for an exam, master's and PhD students analysing thesis data, and analysts who > need a quick, correct comparison of means. It speeds up learning by putting the decisions in order (which test, which > checks, which effect size, which sentence to report), by showing one example fully solved, one half solved and one > left for you, and by ending with self-test questions whose answers sit in a separate key. The worked examples use > fictional data; replace them with your own. > Source basis: Welch (1947), Student (1908), Cohen (1988), the NIST/SEMATECH e-Handbook of Statistical Methods, and the > R and SciPy documentation, all written up in original words and examples. ## Document control | Field | Value | |---|---| | Course or project | [STAT 101 / thesis chapter 4 / client project] | | Learner or analyst | [Your name] | | Data set | [Lakemoor University STAT 101 exam scores (fictional) / your data set and source] | | Software and versions | R 4.5.1 (stats); Python 3.14 with numpy 2.5.4, scipy 1.18.1, pandas 3.1.0 | | Date | YYYY-MM-DD | | Version | [0.1] | ## When to use it **One-sentence rule:** use a t-test when you want to compare one mean, or the difference between two means, for a continuous outcome, and the sample means can be treated as approximately normal (because the data are roughly normal or the samples are large enough). There are three versions, and the design decides which one you use: - **One-sample t-test:** one group, compared with a fixed reference value (a target, a published norm, a specification). - **Independent two-sample t-test:** two separate groups of different people or units. Use **Welch's t-test** by default; it does not assume equal variances and loses almost nothing when the variances happen to be equal (Delacre, Lakens and Leys 2017). Use the pooled (Student) version only when equal variances are justified by design or prior knowledge, or when a course or protocol requires it. - **Paired t-test:** two measurements on the same units (before and after, left and right, matched pairs). It is a one-sample t-test on the differences. | Use when | Do not use when | |---|---| | Outcome is continuous (interval or ratio), e.g. exam score, time, weight | Outcome is binary or a proportion (use a proportion test) | | One or two groups, or two linked measurements | Three or more groups (use one-way ANOVA, not many t-tests) | | Observations are independent within each group | Units are clustered (students within classes) and the clustering is ignored | | Data roughly symmetric, or each group has n of about 30 or more (a rule of thumb, not a guarantee) | Small samples with strong skew or extreme outliers (use a rank test or bootstrap) | | You care about the mean | You care about the median or the whole distribution | **Alternatives:** Mann-Whitney U or Wilcoxon signed-rank (`academic/stats-nonparametric-tests`); permutation test or bootstrap CI for the mean difference (`academic/stats-resampling-methods`); one-way ANOVA or Welch ANOVA for three or more groups (`academic/stats-one-way-anova`); linear regression when you need to adjust for covariates (`academic/stats-linear-regression`); mixed models for clustered or repeated data (`academic/stats-mixed-models`). ## The question and the data **Research question (example):** At Lakemoor University (fictional), STAT 101 ran two sections in one term. One section was taught by traditional lecture, the other with a flipped classroom (videos at home, problem solving in class). Do the final exam scores differ between the two sections, and by how much? | Variable | Role | Measurement level | Values | |---|---|---|---| | `score` | Outcome | Ratio (0 to 100 points) | Final exam score | | `section` | Grouping | Nominal, 2 levels | lecture, flipped | | Design | Independent groups | Different students in each section | n = 10 and n = 12 | The worked example data set (fictional): | Section | Scores | n | Mean | SD | |---|---|---|---|---| | Lecture | 62, 70, 58, 75, 66, 71, 64, 69, 60, 68 | 10 | 66.30 | 5.31 | | Flipped | 74, 81, 66, 90, 72, 85, 79, 68, 88, 76, 83, 70 | 12 | 77.67 | 7.92 | Look at the data before you test anything. The chart shows each student as a dot and each section mean as a thick tick. The flipped section sits higher, and it is also more spread out: that unequal spread is exactly why Welch's version is the safer default. ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "description": "Lakemoor University STAT 101 final exam scores by section (fictional data)", "width": 380, "height": 140, "data": {"values": [ {"section": "Lecture", "score": 62}, {"section": "Lecture", "score": 70}, {"section": "Lecture", "score": 58}, {"section": "Lecture", "score": 75}, {"section": "Lecture", "score": 66}, {"section": "Lecture", "score": 71}, {"section": "Lecture", "score": 64}, {"section": "Lecture", "score": 69}, {"section": "Lecture", "score": 60}, {"section": "Lecture", "score": 68}, {"section": "Flipped", "score": 74}, {"section": "Flipped", "score": 81}, {"section": "Flipped", "score": 66}, {"section": "Flipped", "score": 90}, {"section": "Flipped", "score": 72}, {"section": "Flipped", "score": 85}, {"section": "Flipped", "score": 79}, {"section": "Flipped", "score": 68}, {"section": "Flipped", "score": 88}, {"section": "Flipped", "score": 76}, {"section": "Flipped", "score": 83}, {"section": "Flipped", "score": 70} ]}, "encoding": { "y": {"field": "section", "type": "nominal", "title": null}, "x": {"field": "score", "type": "quantitative", "title": "Final exam score", "scale": {"domain": [50, 95]}} }, "layer": [ {"mark": {"type": "point", "filled": true, "opacity": 0.6, "size": 60}}, {"mark": {"type": "tick", "thickness": 3, "size": 30, "color": "black"}, "encoding": {"x": {"aggregate": "mean", "field": "score", "type": "quantitative"}}} ] } ``` ## Assumptions and how to check each The t-test is robust to mild non-normality, especially with balanced groups of moderate size. It is not robust to dependence between observations, and it is sensitive to extreme outliers. Formal normality tests have little power in small samples and flag trivial departures in large ones, so treat them as a supplement to a plot, never as the decision rule. | Assumption | How to check | R | Python | If violated | |---|---|---|---|---| | Independence of observations | Design: random sampling or assignment, no clustering, no repeated rows per person | Read the protocol | Read the protocol | Use a paired test, a mixed model, or cluster-robust methods | | Correct pairing (paired test only) | Each "before" row matches its "after" row by ID | `stopifnot(all(df$id_before == df$id_after))` | `assert (a.id == b.id).all()` | Fix the merge before testing | | Approximate normality of the means | Dot plot or Q-Q plot per group (of the differences for a paired test); Shapiro-Wilk as a supplement | `qqnorm(x); shapiro.test(x)` | `stats.probplot(x, plot=ax); stats.shapiro(x)` | Larger n helps (central limit theorem); otherwise rank test, bootstrap or transform | | No extreme outliers | Boxplot; check each flagged value against the source | `boxplot(score ~ section)` | `df.boxplot("score", by="section")` | Correct data errors; report with and without true outliers; robust or rank methods | | Equal variances (pooled test only) | Compare SDs; ratio of larger to smaller variance | `var.test(a, b)` (F-test) | `stats.levene(a, b)` (Levene, a different test; there is no SciPy F-test function) | Use Welch (the default anyway) | In Worked example A, Shapiro-Wilk gives p = .965 (lecture) and p = .829 (flipped), the dot plot shows no extreme values, and the variance ratio is 62.79 / 28.23 = 2.22 (F-test p = .240). An F-test that is "not significant" does not prove equal variances, which is one more reason not to let a pre-test switch you between the pooled and Welch versions. Use Welch and move on. ## How it works Every t-statistic is a **signal-to-noise ratio**. The signal is how far the observed mean (or difference in means) sits from the value claimed by the null hypothesis. The noise is the standard error SE: how much that mean would wobble from sample to sample by chance alone. If t is large in absolute value, the observed difference is large compared with the chance wobble, and the p-value is small. The t distribution (not the normal) is used because the SE itself is estimated from the data; with few degrees of freedom df, the t distribution has heavier tails to account for that extra uncertainty. **One-sample t-test** (hypothesised mean $\mu_0$): $$ t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}}, \qquad df = n - 1 $$ **Paired t-test** (differences $d_i = x_{i,\text{after}} - x_{i,\text{before}}$): a one-sample test on the $d_i$ with $\mu_0 = 0$: $$ t = \frac{\bar{d}}{s_d/\sqrt{n}}, \qquad df = n - 1 $$ **Welch's independent t-test** (default): $$ t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}}, \qquad df_{W} = \frac{\left(\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}\right)^2} {\dfrac{(s_1^2/n_1)^2}{n_1 - 1} + \dfrac{(s_2^2/n_2)^2}{n_2 - 1}} $$ The df formula is the Welch-Satterthwaite approximation. It usually gives a non-integer df that lies between the smaller of $n_1 - 1$ and $n_2 - 1$ and the pooled value $n_1 + n_2 - 2$. **Pooled (Student) independent t-test** (only when equal variances are justified): $$ s_p = \sqrt{\frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}}, \qquad t = \frac{\bar{x}_1 - \bar{x}_2}{s_p\sqrt{\dfrac{1}{n_1} + \dfrac{1}{n_2}}}, \qquad df = n_1 + n_2 - 2 $$ **Confidence interval** for any of the three: estimate $\pm\ t_{0.975,\,df} \times SE$. **Effect sizes:** Cohen's d for two independent groups uses the pooled SD; Hedges' g applies a small-sample correction; for paired data, $d_z$ divides the mean difference by the SD of the differences. $$ d = \frac{\bar{x}_1 - \bar{x}_2}{s_p}, \qquad g = J\,d, \quad J = 1 - \frac{3}{4(n_1 + n_2) - 9}, \qquad d_z = \frac{\bar{d}}{s_d} $$ An approximate standard error for d (used for a normal-approximation 95% CI) is $SE_d = \sqrt{\dfrac{n_1 + n_2}{n_1 n_2} + \dfrac{d^2}{2(n_1 + n_2)}}$; multiply the CI limits by $J$ for g. | Symbol | Meaning | |---|---| | $\bar{x}$, $\bar{x}_1$, $\bar{x}_2$ | Sample mean (of group 1, group 2) | | $\mu_0$ | Value of the mean under the null hypothesis | | $s$, $s_1$, $s_2$ | Sample standard deviation (divisor $n - 1$) | | $n$, $n_1$, $n_2$ | Sample size n | | $d_i$, $\bar{d}$, $s_d$ | Paired difference, its mean and its SD | | $s_p$ | Pooled standard deviation | | $SE$ | Standard error of the mean or of the difference | | $df$, $df_W$ | Degrees of freedom, Welch-Satterthwaite degrees of freedom | | $d$, $g$, $d_z$, $J$ | Cohen's d, Hedges' g, paired effect size, small-sample correction | ```mermaid flowchart TD A["Continuous outcome, comparing means"] --> B{"How many groups or measurements?"} B -->|"One group vs a fixed value"| C["One-sample t-test"] B -->|"Two measurements on the same units"| D["Paired t-test on the differences"] B -->|"Two separate groups"| E{"Equal variances justified by design?"} E -->|"No or unsure (default)"| F["Welch t-test"] E -->|"Yes, and protocol requires it"| G["Pooled Student t-test"] B -->|"Three or more groups"| H["One-way ANOVA or Welch ANOVA"] C --> I{"Strong skew or outliers with small n?"} D --> I F --> I G --> I I -->|"Yes"| J["Rank test or bootstrap"] I -->|"No"| K["Report t, df, p, mean difference, 95% CI, effect size"] ``` ## Worked example A (fully solved): Welch and pooled independent t-tests **Hypotheses.** Let $\mu_F$ and $\mu_L$ be the population mean exam scores under the flipped and lecture formats. $H_0: \mu_F - \mu_L = 0$ versus $H_1: \mu_F - \mu_L \neq 0$ (two-sided), α = 0.05. **Checks.** Independent groups by design (different students, one score each). No extreme outliers in the dot plot. Shapiro-Wilk p = .965 and p = .829 (no evidence against normality, but with n = 10 and 12 this test is weak). SDs 7.92 versus 5.31: a variance ratio of 2.22, so Welch is clearly the right default. **R code and output.** R's `t.test()` uses Welch unless you ask for `var.equal = TRUE`. ```r # Lakemoor University (fictional): STAT 101 final exam scores lecture <- c(62, 70, 58, 75, 66, 71, 64, 69, 60, 68) flipped <- c(74, 81, 66, 90, 72, 85, 79, 68, 88, 76, 83, 70) c(mean(lecture), sd(lecture), mean(flipped), sd(flipped)) shapiro.test(lecture)$p.value; shapiro.test(flipped)$p.value var.test(flipped, lecture)$p.value t.test(flipped, lecture) # Welch (R default, var.equal = FALSE) t.test(flipped, lecture, var.equal = TRUE) # pooled Student, for comparison # Cohen's d (pooled SD), Hedges' g and an approximate 95% CI for g n1 <- length(flipped); n2 <- length(lecture) sp <- sqrt(((n1 - 1) * var(flipped) + (n2 - 1) * var(lecture)) / (n1 + n2 - 2)) d <- (mean(flipped) - mean(lecture)) / sp J <- 1 - 3 / (4 * (n1 + n2) - 9) g <- J * d se_d <- sqrt((n1 + n2) / (n1 * n2) + d^2 / (2 * (n1 + n2))) round(c(sp = sp, d = d, g = g, lo = g - 1.96 * J * se_d, hi = g + 1.96 * J * se_d), 3) ``` ```text [1] 66.300000 5.313505 77.666667 7.923880 [1] 0.9654593 [1] 0.8294417 [1] 0.2402227 Welch Two Sample t-test data: flipped and lecture t = 4.0048, df = 19.23, p-value = 0.0007426 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: 5.430958 17.302376 sample estimates: mean of x mean of y 77.66667 66.30000 Two Sample t-test data: flipped and lecture t = 3.8625, df = 20, p-value = 0.0009702 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: 5.227991 17.505342 sp d g lo hi 6.873 1.654 1.591 0.657 2.525 ``` **Python code and output.** Here is the key teaching point of this card: **SciPy's `ttest_ind()` defaults to `equal_var=True`, the pooled test, while R's `t.test()` defaults to Welch.** Run the same call in both languages with default arguments and you get two different answers (t = 3.86 with df = 20 in SciPy, t = 4.00 with df = 19.23 in R). Always set `equal_var=False` in Python unless you have a reason not to. ```python import numpy as np from scipy import stats lecture = np.array([62, 70, 58, 75, 66, 71, 64, 69, 60, 68]) flipped = np.array([74, 81, 66, 90, 72, 85, 79, 68, 88, 76, 83, 70]) print(stats.ttest_ind(flipped, lecture)) # SciPy default: pooled res = stats.ttest_ind(flipped, lecture, equal_var=False) # Welch print(res) print(res.confidence_interval(0.95)) ``` ```text TtestResult(statistic=np.float64(3.8624700393746134), pvalue=np.float64(0.0009702033494507789), df=np.float64(20.0)) TtestResult(statistic=np.float64(4.004816788599783), pvalue=np.float64(0.0007426031571542732), df=np.float64(19.23042884691189)) ConfidenceInterval(low=np.float64(5.430957692222952), high=np.float64(17.302375641110395)) ``` **Calculation by hand (Welch).** Group 1 is flipped ($n_1 = 12$, $\bar{x}_1 = 77.667$, $s_1^2 = 62.788$), group 2 is lecture ($n_2 = 10$, $\bar{x}_2 = 66.300$, $s_2^2 = 28.233$). 1. Variance of each mean: $s_1^2/n_1 = 62.788/12 = 5.2323$ and $s_2^2/n_2 = 28.233/10 = 2.8233$. 2. Standard error: $SE = \sqrt{5.2323 + 2.8233} = \sqrt{8.0556} = 2.8382$. 3. Statistic: $t = (77.667 - 66.300)/2.8382 = 11.367/2.8382 = 4.005$. 4. Welch-Satterthwaite df: numerator $8.0556^2 = 64.893$; denominator $5.2323^2/11 + 2.8233^2/9 = 2.4888 + 0.8857 = 3.3745$; so $df_W = 64.893/3.3745 = 19.23$. 5. Critical value $t_{0.975,\,19.23} = 2.091$, so the 95% CI is $11.367 \pm 2.091 \times 2.8382 = [5.43,\ 17.30]$. The hand result matches both programs. Notice that the pooled test, with $df = 20$, gives a slightly smaller t here: the pooled variance (47.24) replaces both group variances, so the smaller lecture group (own variance 28.23, n = 10) is credited with a much larger variance, and the SE rises from 2.84 to 2.94. When the smaller group has the larger variance, the effect reverses and the pooled test becomes too liberal. **Effect size.** Cohen's d = 11.367 / 6.873 = 1.65; Hedges' g = 1.59, approximate 95% CI [0.66, 2.53] (normal approximation with the SE formula above; exact intervals based on the noncentral t distribution differ slightly). That is a large difference by Cohen's rough benchmarks, but the CI is wide because the samples are small: the true effect could be moderate or very large. **Interpretation.** The flipped section scored on average 11.4 points higher than the lecture section. A difference this large would be rare if the two formats had the same population mean (p < .001). Because students were not randomly assigned to sections, this is evidence of a difference between the two sections, not proof that the flipped format caused it. **APA-style report.** Students in the flipped section (*M* = 77.67, *SD* = 7.92, *n* = 12) scored higher than students in the lecture section (*M* = 66.30, *SD* = 5.31, *n* = 10), Welch's *t*(19.23) = 4.00, *p* < .001, mean difference = 11.37 points, 95% CI [5.43, 17.30], Hedges' *g* = 1.59, 95% CI [0.66, 2.53]. ## Worked example B (half solved): paired t-test **Setting.** Ten Lakemoor University students took a 20-point statistics quiz before and after a two-hour revision workshop run by the campus learning centre. The same students were measured twice, so the design is paired. | Student | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |---|---|---|---|---|---|---|---|---|---|---| | Before | 12 | 15 | 9 | 14 | 11 | 13 | 10 | 16 | 12 | 8 | | After | 14 | 17 | 10 | 15 | 14 | 13 | 13 | 18 | 13 | 11 | | Difference (after minus before) | 2 | 2 | 1 | 1 | 3 | 0 | 3 | 2 | 1 | 3 | ```r before <- c(12, 15, 9, 14, 11, 13, 10, 16, 12, 8) after <- c(14, 17, 10, 15, 14, 13, 13, 18, 13, 11) d <- after - before mean(d); sd(d); shapiro.test(d)$p.value t.test(after, before, paired = TRUE) ``` ```python import numpy as np from scipy import stats before = np.array([12, 15, 9, 14, 11, 13, 10, 16, 12, 8]) after = np.array([14, 17, 10, 15, 14, 13, 13, 18, 13, 11]) r = stats.ttest_rel(after, before) print(r, r.confidence_interval()) ``` ```text # Trimmed and combined output from the two blocks above R: mean(d) = 1.8 sd(d) = 1.032796 Shapiro-Wilk p = 0.191 t = 5.5114, df = 9, p-value = 0.0003746 95 percent confidence interval: 1.061183 2.538817 Python: TtestResult(statistic=np.float64(5.511351921262151), pvalue=np.float64(0.0003746456530901087), df=np.int64(9)) ConfidenceInterval(low=np.float64(1.0611825644216402), high=np.float64(2.53881743557836)) ``` Complete the steps: 1. Hypotheses: $H_0: \mu_d = $ [ ] versus $H_1:$ [ ]. 2. Standard error of the mean difference: $SE = s_d/\sqrt{n} = 1.0328/\sqrt{10} = $ [ ]. 3. Statistic: $t = \bar{d}/SE = 1.8/$[ ] $=$ [ ], with df = [ ]. 4. 95% CI: $1.8 \pm t_{0.975,\,9} \times SE = 1.8 \pm 2.262 \times$ [ ] $=$ [ ]. 5. Effect size: $d_z = \bar{d}/s_d = $ [ ]. 6. Report: Quiz scores rose after the workshop, mean difference = [ ] points, *t*([ ]) = [ ], *p* [ ], 95% CI [ ], $d_z$ = [ ]. Now the trap. If you analyse these data with an independent test by mistake (`t.test(after, before)`), you get t = 1.60 and p = .127: "not significant". The pairing removes the large between-student differences (the before and after scores correlate at r = 0.92), which makes the SE of the paired difference much smaller. Using the wrong design throws that information away. ## Exercise C (unsolved): one-sample t-test Lakemoor University recommends that first-year students study 10 hours per week outside class for STAT 101. Twelve students logged their study hours for one typical week: | Student | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Hours | 8.5 | 11.0 | 7.5 | 9.0 | 12.5 | 6.0 | 10.0 | 8.0 | 9.5 | 7.0 | 11.5 | 8.5 | Tasks: 1. State $H_0$ and $H_1$ (two-sided, α = 0.05). 2. Compute the mean, SD, SE, t and df by hand. ($t_{0.975,\,11} = 2.201$.) 3. Compute the 95% CI for the mean and the effect size $d = (\bar{x} - \mu_0)/s$. 4. Run `t.test(hours, mu = 10)` in R and `stats.ttest_1samp(hours, 10)` in Python to check your answers. 5. Write a one-sentence APA-style result and a one-sentence plain-language conclusion. Does a non-significant result show that students study 10 hours on average? ## Reading software output | Output line (R `t.test`) | Python equivalent | Meaning | Report it? | |---|---|---|---| | `Welch Two Sample t-test` | `equal_var=False` in the call | Which test ran. If it says `Two Sample t-test` (no Welch), the pooled test ran | Name the test | | `t = 4.0048` | `statistic` | Signal-to-noise ratio; sign depends on the order of the groups | Yes, two decimals | | `df = 19.23` | `df` | Degrees of freedom; non-integer means Welch | Yes, two decimals for Welch | | `p-value = 0.0007426` | `pvalue` | Probability of a t at least this extreme if $H_0$ were true | Yes: three decimals or p < .001 | | `alternative hypothesis: ... not equal to 0` | `alternative='two-sided'` | Two-sided test (the default in both) | Only if one-sided | | `95 percent confidence interval: 5.43 17.30` | `confidence_interval()` | Plausible range for the true mean difference | Yes, always | | `mean of x mean of y` | compute with `.mean()` | Group means (R does not print the difference for two-sample tests) | Report means, SDs and the difference | ## Reporting template **Fill-in sentence (independent, Welch):** [Group 1] (*M* = [ ], *SD* = [ ], *n* = [ ]) scored [higher / lower] than [Group 2] (*M* = [ ], *SD* = [ ], *n* = [ ]) on [outcome], Welch's *t*([df]) = [t], *p* = [p], mean difference = [ ] [units], 95% CI [[lower], [upper]], Hedges' *g* = [ ], 95% CI [[ ], [ ]]. **Fill-in sentence (paired):** [Outcome] changed from *M* = [ ] (*SD* = [ ]) at [time 1] to *M* = [ ] (*SD* = [ ]) at [time 2], mean change = [ ], 95% CI [[ ], [ ]], *t*([n - 1]) = [ ], *p* = [ ], $d_z$ = [ ]. **Fill-in sentence (one-sample):** The mean [outcome] (*M* = [ ], *SD* = [ ], *n* = [ ]) [did / did not] differ from the reference value of [μ0], *t*([n - 1]) = [ ], *p* = [ ], mean difference = [ ], 95% CI [[ ], [ ]], *d* = [ ]. | Group or condition | n | Mean | SD | Mean difference [95% CI] | t (df) | p | Effect size [95% CI] | |---|---|---|---|---|---|---|---| | *Example:* Flipped vs lecture | 12 / 10 | 77.67 / 66.30 | 7.92 / 5.31 | 11.37 [5.43, 17.30] | 4.00 (19.23) | < .001 | g = 1.59 [0.66, 2.53] | | [Your comparison] | [ ] | [ ] | [ ] | [ ] | [ ] | [ ] | [ ] | **Results checklist:** - [ ] The design (independent, paired, one-sample) is stated and matches how the data were collected. - [ ] The test version is named (Welch or pooled) and the software default was checked. - [ ] Means, SDs and n for each group or time point are reported. - [ ] t, df, exact p (or p < .001) and the direction of the difference are reported. - [ ] The mean difference with its 95% CI is reported in the original units. - [ ] An effect size with a CI is reported and its formula (d, g or $d_z$) is named. - [ ] Assumption checks and any outliers or exclusions are described. - [ ] One-sided tests, if used, were planned in advance and are labelled. - [ ] The conclusion matches the design (association versus causation). ## Common mistakes and misconceptions - **Trusting software defaults across languages.** R `t.test()` defaults to Welch (`var.equal = FALSE`); SciPy `ttest_ind()` defaults to the pooled test (`equal_var=True`). The same data give different t, df and p. Always set the argument explicitly and say which test you ran. - **Using a variance pre-test to choose between pooled and Welch.** Two-stage procedures (Levene first, then pick a t-test) distort the Type I error rate. Use Welch unconditionally. - **Analysing paired data as independent.** In Worked example B this changes p from < .001 to .127. Pairing is decided by the design, not by which test gives the smaller p-value. - **Running many t-tests for three or more groups.** With four groups there are six pairwise tests; the chance of at least one false positive grows well above 5%. Use ANOVA with planned contrasts or a post hoc procedure. - **"p = 0.03 means a 3% chance that H0 is true."** No. The p-value is the probability of data at least this extreme if H0 were true. It says nothing directly about the probability of H0 (see the ASA statement on p-values). - **"Not significant means no difference."** A non-significant result with a wide CI is inconclusive; the CI may contain both zero and large effects. Absence of evidence is not evidence of absence (see equivalence testing in further reading). - **Testing normality of the raw pooled data.** For two groups, check each group separately; for a paired test, check the differences, not the before and after scores. - **Reporting d without saying which d.** Cohen's d with pooled SD, Hedges' g and $d_z$ for paired data can differ a lot for the same data. Name the formula. - **Ignoring clustering.** Students nested in tutorial groups are not independent. A t-test on individual students then understates the SE; use a mixed model (`academic/stats-mixed-models`). - **Claiming causation from non-randomised groups.** Sections that students choose themselves can differ in prior ability. Only random assignment supports a causal reading. ## Self-test questions 1. A colleague runs `scipy.stats.ttest_ind(a, b)` and you run `t.test(a, b)` in R on the same data. Why might your results differ, and how do you make them agree on Welch? 2. Why does Welch's test usually give a non-integer df, and between which two values does it lie? 3. A study measures blood pressure in 15 patients before and after a diet. Which t-test applies, and on what variable do you check normality? 4. Two groups give t(38) = 2.10, p = .042, d = 0.66, 95% CI for d [0.02, 1.30]. What does the CI tell you that the p-value does not? 5. Interleaving check: for each design, name the test. (a) Exam scores of 40 students compared with a national mean of 65. (b) Reaction times of 18 participants under two lighting conditions, each person tested in both. (c) Weekly sales of Brightwell Bakery's two shops over 20 weeks, where the question is which shop sells more on average. 6. Why is Hedges' g preferred over Cohen's d in small samples? ## My notes and next steps | Item | My entry | |---|---| | Confidence with t-tests (1 = lost, 5 = could teach it) | [ ] | | Which test I chose for my own data and why | [ ] | | Open questions to ask my tutor or supervisor | [ ] | | Mistakes logged in `academic/stats-error-log` | [Date and short description] | | Next review date (spacing: +2 days, +1 week, +1 month) | YYYY-MM-DD | | Next template to study | [academic/stats-nonparametric-tests / academic/stats-one-way-anova] | ## Sources and further reading Structure aligned with the NIST/SEMATECH e-Handbook of Statistical Methods (sections on two-sample t-tests) and with the reporting elements named in APA JARS-Quant; all wording, data and examples here are original. - Student (1908). The probable error of a mean. *Biometrika*, 6(1), 1–25. - Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. *Biometrika*, 34(1–2), 28–35. - Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. *Biometrics Bulletin*, 2(6), 110–114. - Cohen, J. (1988). *Statistical Power Analysis for the Behavioral Sciences* (2nd ed.). Lawrence Erlbaum. - Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. *Journal of Educational Statistics*, 6(2), 107–128. - Delacre, M., Lakens, D., and Leys, C. (2017). Why psychologists should by default use Welch's t-test instead of Student's t-test. *International Review of Social Psychology*, 30(1), 92–101. - Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. *The American Statistician*, 70(2), 129–133. - NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.5.3 Two-Sample t-Test for Equal Means, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain. - R Core Team. `t.test` documentation, package stats, R 4.5.1. SciPy documentation for `scipy.stats.ttest_ind`, `ttest_rel` and `ttest_1samp`, SciPy 1.18. - Further reading (by name only): de Smith, M. J., *Statistical Analysis Handbook* (2024 edition, statsref.com); OpenIntro Statistics; American Psychological Association, *Publication Manual* (7th ed.) and JARS-Quant; Lakens, D. (2017), equivalence tests (TOST), *Social Psychological and Personality Science*, 8(4), 355–362. ## Answer key **Worked example B.** (1) $H_0: \mu_d = 0$ versus $H_1: \mu_d \neq 0$. (2) $SE = 1.0328/3.1623 = 0.3266$. (3) $t = 1.8/0.3266 = 5.51$, df = 9. (4) $1.8 \pm 2.262 \times 0.3266 = 1.8 \pm 0.739$, so 95% CI [1.06, 2.54]. (5) $d_z = 1.8/1.0328 = 1.74$. (6) Quiz scores rose after the workshop, mean difference = 1.80 points, *t*(9) = 5.51, *p* < .001, 95% CI [1.06, 2.54], $d_z$ = 1.74. The differences look roughly normal (Shapiro-Wilk p = .191) and there are no extreme values. **Exercise C.** (1) $H_0: \mu = 10$ versus $H_1: \mu \neq 10$. (2) $\bar{x} = 9.083$, $s = 1.917$, $SE = 1.917/\sqrt{12} = 0.553$, $t = (9.083 - 10)/0.553 = -1.66$, df = 11. (3) 95% CI $9.083 \pm 2.201 \times 0.553 = [7.87,\ 10.30]$; $d = -0.917/1.917 = -0.48$. (4) R and Python both give t = -1.6566, df = 11, p = 0.1258, 95% CI [7.865, 10.301]. (5) "Mean weekly study time (*M* = 9.08 hours, *SD* = 1.92, *n* = 12) did not differ significantly from the recommended 10 hours, *t*(11) = -1.66, *p* = .126, 95% CI [7.87, 10.30], *d* = -0.48." In plain words: the sample studied a little under 10 hours, but with 12 students we cannot rule out that the true average is 10. No, the result does not show that students study 10 hours: the CI runs from about 7.9 to 10.3 hours, so an average well below the target is also plausible. A larger sample (see `academic/stats-power-sample-size`) or an equivalence test would be needed. **Self-test.** 1. SciPy's `ttest_ind` defaults to the pooled test (`equal_var=True`); R's `t.test` defaults to Welch (`var.equal = FALSE`). Use `ttest_ind(a, b, equal_var=False)` in Python, or keep the R default. 2. The Welch-Satterthwaite formula weights each group's variance of the mean, so df is a continuous value. It lies between $\min(n_1 - 1, n_2 - 1)$ and $n_1 + n_2 - 2$; it reaches the upper bound when both the sample variances and the sample sizes are equal. 3. Paired t-test (same patients measured twice). Check normality of the 15 differences, not of the raw before and after values. 4. The CI shows the effect is estimated very imprecisely: the data are consistent with a negligible effect (0.02) and with a very large one (1.30). The p-value only says zero is (just) outside the 95% interval. 5. (a) One-sample t-test against 65. (b) Paired t-test (each person in both conditions). (c) Two independent groups (Welch), but check whether the weeks are paired: if both shops are observed in the same 20 weeks, a paired test on weekly differences removes shared week-to-week variation, and autocorrelation over time should also be checked. 6. Cohen's d overestimates the population effect in small samples; Hedges' correction factor $J$ (slightly below 1) removes most of that bias. With n = 22 in Worked example A, d = 1.65 becomes g = 1.59. ## Revision history | Version | Date | Author | Change summary | |---|---|---|---| | 0.1 | YYYY-MM-DD | [Your name] | First draft from the template | | 0.2 | YYYY-MM-DD | [Your name] | *Example:* Replaced Lakemoor University example with thesis data, Welch test run |