# Effect Sizes: [Your study question or course] > A working reference and practice sheet for effect sizes: how to choose one for your design, how to compute it by > hand and in R and Python, how to convert between effect size families, how to put a confidence interval around it, > and how to interpret it without leaning on "small, medium, large" labels. It is for statistics students preparing > for an exam, master's and PhD students reporting thesis results, and analysts who need to say how big an effect is, > not only whether it exists. It speeds up learning with a decision diagram, a fully solved example, a half solved > example, an exercise, self-test questions and a separate answer key. The worked examples use fictional data; > replace them with your own. > Source basis: standard effect size formulas (Cohen 1988; Hedges 1981; Hedges and Olkin 1985; Lakens 2013) written > in original notation and words. ## Document control This table records the course or project the sheet supports, the data set behind the numbers and the software that produced them. Rows marked *Example:* show how to fill a field. | Field | Value | |---|---| | Course or project | [STAT 201 Applied Statistics / thesis chapter 4] | | Learner or analyst | [Your name] | | Data set | [Name, source, date extracted] *Example:* Lakemoor University revision study (fictional), 20 students | | Software and versions | R 4.5.1 (stats); Python 3.14, numpy 2.5.4, pandas 3.1.0, scipy 1.18.1, statsmodels 0.15.0 | | Date | YYYY-MM-DD | | Version | [0.1] | ## What an effect size is and why it matters A hypothesis test answers a narrow question: is the observed pattern surprising if there were no effect at all? An effect size answers the question your reader actually cares about: how big is the effect? A p-value mixes the size of the effect with the sample size n, so a trivial effect becomes "significant" with enough data and a large effect can miss significance in a small study. The effect size separates the two. It is the number that goes into a meta-analysis, the number a power analysis needs, and the number a decision maker uses to judge whether a result is worth acting on. Effect sizes come in two kinds. **Unstandardized** effect sizes keep the original units: "students who revised with flashcards scored 5.4 points higher", "the new yeast raised loaves by 0.7 cm". When the units are meaningful, these are usually the best thing to report, because anyone can judge whether 5.4 exam points matters. **Standardized** effect sizes divide by a measure of variability or express the effect as a share of variance: Cohen's d, r, η², Cramér's V. They let you compare studies that used different scales and they feed power calculations. Report the unstandardized effect whenever the scale has meaning, and add a standardized one when you need comparability. Every effect size is an estimate from a sample, so it has sampling error. A d of 1.10 from 20 students is compatible with true effects from small to very large. That is why every effect size you report should carry a 95% confidence interval (CI). ## Choosing an effect size The right effect size follows from three facts about your study: the type of outcome (continuous, binary, counts in categories), the number of groups or predictors, and whether the measurements are independent or paired. The diagram below walks through those decisions; the table that follows gives the default choice and a common alternative. ```mermaid flowchart TD A["What kind of outcome?"] --> B["Continuous"] A --> C["Categorical (counts)"] B --> D["How many groups or predictors?"] D --> E["Two groups"] D --> F["Three or more groups"] D --> G["Continuous predictor(s)"] E --> H["Independent?"] H -->|Yes| I["Cohen's d or Hedges' g"] H -->|"No, paired"| J["d_z for the test, d_av for comparison"] F --> K["eta squared, omega squared, partial eta squared"] G --> L["One predictor: r"] G --> M["Several predictors: R squared and f squared"] C --> N["Table size?"] N -->|"2 x 2"| O["Odds ratio, risk ratio, phi"] N -->|"Larger"| P["Cramer's V"] ``` | Design | Default effect size | Common alternative | Notes | |---|---|---|---| | Two independent groups, continuous outcome | Hedges' g (bias-corrected d) | Mean difference in raw units; Glass's Δ when one group is a control with a different SD | Report the raw difference too | | Two paired measurements | d_z (for power and the test) and d_av (to compare with between-subject studies) | Raw mean change | Always say which d you used | | Three or more groups (one factor) | ω² | η² | η² is biased upward in small samples | | Factorial or multi-factor design | Partial η² or partial ω² for each effect | Generalized η² (Olejnik and Algina 2003) for comparison across designs | Partial values cannot be added up | | Two continuous variables | Pearson r (Spearman ρ for ranks) | r² | r is already standardized | | Multiple regression | R² and adjusted R² for the model; f² for a predictor set | Standardized coefficients | f² for a single added predictor is the useful one | | 2 x 2 table | Odds ratio (case-control, logistic) or risk ratio (cohort, trial) | φ, risk difference | Risk ratio is easier to explain | | Larger contingency table | Cramér's V | Odds ratios for chosen pairs of cells | V depends on table size | *Example:* the Lakemoor revision study compares exam scores (continuous) between two independent groups of students, so the default is Hedges' g, reported with the raw mean difference. ## The data used in the examples All examples use small fictional data sets typed into the code, so nothing needs to be downloaded. | Data set | Design | Variables | n | |---|---|---|---| | Lakemoor revision study | Two independent groups | Exam score (0 to 100) after flashcards or re-reading | 10 + 10 | | Lakemoor anxiety workshop | Paired, before and after | Test anxiety score (higher = more anxious) | 8 | | Brightwell Bakery yeast trial | One factor, three levels | Loaf height (cm) for yeast strains A, B, C | 6 per strain | | Lakemoor study hours | Two continuous variables, plus a second predictor | Hours studied, hours of sleep, exam score | 12 | | Lakemoor revision sessions | 2 x 2 table | Attended sessions (yes or no) x passed (yes or no) | 80 | The revision study scores are: flashcards 72, 75, 81, 66, 79, 70, 74, 84, 77, 71 (mean 74.9, SD 5.466); re-reading 65, 70, 74, 68, 72, 63, 77, 71, 66, 69 (mean 69.5, SD 4.249). The chart below shows every score so you can see the overlap that a single d value hides. ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "description": "Exam scores by revision method, Lakemoor revision study (fictional)", "width": 360, "height": 140, "data": {"values": [ {"method": "Flashcards", "score": 72}, {"method": "Flashcards", "score": 75}, {"method": "Flashcards", "score": 81}, {"method": "Flashcards", "score": 66}, {"method": "Flashcards", "score": 79}, {"method": "Flashcards", "score": 70}, {"method": "Flashcards", "score": 74}, {"method": "Flashcards", "score": 84}, {"method": "Flashcards", "score": 77}, {"method": "Flashcards", "score": 71}, {"method": "Re-reading", "score": 65}, {"method": "Re-reading", "score": 70}, {"method": "Re-reading", "score": 74}, {"method": "Re-reading", "score": 68}, {"method": "Re-reading", "score": 72}, {"method": "Re-reading", "score": 63}, {"method": "Re-reading", "score": 77}, {"method": "Re-reading", "score": 71}, {"method": "Re-reading", "score": 66}, {"method": "Re-reading", "score": 69} ]}, "mark": {"type": "tick", "thickness": 2}, "encoding": { "x": {"field": "score", "type": "quantitative", "scale": {"domain": [60, 90]}, "title": "Exam score"}, "y": {"field": "method", "type": "nominal", "title": null}, "color": {"field": "method", "type": "nominal", "legend": null} } } ``` ## Formulas: mean differences **Cohen's d** says how many standard deviations apart two group means are. It divides the difference in means by the pooled standard deviation, which is a weighted average of the two group SDs. $$ d = \frac{\bar{x}_1 - \bar{x}_2}{s_p}, \qquad s_p = \sqrt{\frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}} $$ **Hedges' g** removes the small upward bias of d. The exact correction factor uses the gamma function; the approximation below is accurate to three decimals once the degrees of freedom df are above about 10. $$ g = J \cdot d, \qquad J = \frac{\Gamma(m/2)}{\sqrt{m/2}\,\Gamma\!\left(\frac{m-1}{2}\right)} \approx 1 - \frac{3}{4(n_1 + n_2) - 9}, \qquad m = n_1 + n_2 - 2 $$ **Paired designs** have two different "d" values and you must say which one you report. - **d_z** standardizes the mean of the differences by the SD of the differences. It is the effect size the paired t-test actually tests, so it is the right input for a paired power analysis, and it equals the paired t divided by the square root of n. - **d_av** standardizes the mean difference by the average of the two raw SDs. It is on the same scale as an independent-groups d, so use it when you want to compare a within-subject result with between-subject studies. $$ d_z = \frac{\bar{D}}{s_D} = \frac{t}{\sqrt{n}}, \qquad d_{av} = \frac{\bar{D}}{(s_{\text{pre}} + s_{\text{post}})/2} $$ When the two measurements are strongly correlated, s_D is small and d_z is much larger than d_av. With roughly equal SDs the link is $d_z \approx d_{av} / \sqrt{2(1 - r)}$, where r is the correlation between the two measurements. | Symbol | Meaning | |---|---| | $\bar{x}_1, \bar{x}_2$ | Sample means of group 1 and group 2 | | $s_1, s_2$ | Sample standard deviations (n - 1 denominator) | | $n_1, n_2$ | Sample sizes of the two groups | | $s_p$ | Pooled standard deviation | | $J$ | Small-sample correction factor for Hedges' g | | $\bar{D}, s_D$ | Mean and SD of the paired differences | | $r$ | Correlation between the paired measurements (or between two variables) | ## Formulas: variance explained, association and regression **Eta squared (η²)** is the share of the total variation that lies between groups. **Partial η²** divides the effect's sum of squares by itself plus the error sum of squares only, leaving out other effects in the model. In a one-way ANOVA there are no other effects, so η² and partial η² are identical. **Omega squared (ω²)** is a less biased estimate of the population share; it is always smaller than η² and turns negative when F < 1 (report it as 0). $$ \eta^2 = \frac{SS_{\text{effect}}}{SS_{\text{total}}}, \qquad \eta_p^2 = \frac{SS_{\text{effect}}}{SS_{\text{effect}} + SS_{\text{error}}} = \frac{F \cdot df_1}{F \cdot df_1 + df_2} $$ $$ \omega^2 = \frac{SS_{\text{between}} - df_{\text{between}} \cdot MS_{\text{within}}}{SS_{\text{total}} + MS_{\text{within}}} \quad \text{(one-way ANOVA)} $$ **Pearson r** is itself an effect size: the strength of a straight-line relationship, from -1 to 1. Its confidence interval uses Fisher's z transformation, because the sampling distribution of r is skewed when r is far from zero. $$ z = \tanh^{-1}(r), \qquad SE_z = \frac{1}{\sqrt{n - 3}}, \qquad \text{CI} = \tanh\left(z \pm 1.96 \cdot SE_z\right) $$ **Categorical outcomes.** In a 2 x 2 table with cells a, b (first row) and c, d (second row), **φ** is the Pearson correlation between the two binary variables, **Cramér's V** extends it to larger tables, the **odds ratio** compares the odds of the outcome, and the **risk ratio** compares the probabilities. $$ \phi = \frac{ad - bc}{\sqrt{(a+b)(c+d)(a+c)(b+d)}}, \qquad V = \sqrt{\frac{\chi^2}{N \cdot (k - 1)}}, \quad k = \min(\text{rows}, \text{columns}) $$ $$ OR = \frac{a \cdot d}{b \cdot c}, \quad SE_{\ln OR} = \sqrt{\tfrac{1}{a} + \tfrac{1}{b} + \tfrac{1}{c} + \tfrac{1}{d}}; \qquad RR = \frac{a/(a+b)}{c/(c+d)}, \quad SE_{\ln RR} = \sqrt{\tfrac{1}{a} - \tfrac{1}{a+b} + \tfrac{1}{c} - \tfrac{1}{c+d}} $$ Confidence intervals for OR and RR are built on the log scale and exponentiated, so they are not symmetric around the estimate. **Regression.** R² is the share of outcome variance explained by the model. Cohen's f² turns a share of variance into a signal-to-noise ratio and is the input for regression power analysis. The most useful version is the f² for a set of predictors B added to a model that already contains A. $$ f^2 = \frac{R^2}{1 - R^2}, \qquad f^2_{B} = \frac{R^2_{AB} - R^2_{A}}{1 - R^2_{AB}}, \qquad f = \sqrt{\frac{\eta^2}{1 - \eta^2}} \;\text{(ANOVA)} $$ | Symbol | Meaning | |---|---| | $SS$, $MS$, $df$ | Sum of squares, mean square, degrees of freedom from the ANOVA table | | $F$, $df_1$, $df_2$ | F statistic with numerator and denominator df | | $\chi^2$, $N$ | Chi-square statistic and total count in the table | | $a, b, c, d$ | Cell counts of a 2 x 2 table, row by row | | $R^2_{AB}$, $R^2_A$ | R² of the full model and of the model without predictor set B | ## Converting between effect sizes You often need an effect size that a paper did not report, or need to put two studies on one scale. The conversions below are exact for the stated design or are standard approximations (Borenstein et al. 2009). They assume the design is what the formula says: using an independent-groups conversion on a paired t gives a wrong answer. | From | To | Formula | Lakemoor check | |---|---|---|---| | Independent t | d | $d = t\sqrt{1/n_1 + 1/n_2}$ | t = 2.466, n = 10 + 10: d = 1.103 | | Paired t | d_z | $d_z = t / \sqrt{n}$ | t = 4.583, n = 8: d_z = 1.620 | | d | r (point-biserial) | $r = d / \sqrt{d^2 + a}$, $a = (n_1 + n_2)^2 / (n_1 n_2)$ | d = 1.103, a = 4: r = 0.483 | | r | d (equal n) | $d = 2r / \sqrt{1 - r^2}$ | r = 0.483: d = 1.103 | | F (one effect) | partial η² | $F df_1 / (F df_1 + df_2)$ | F(2, 15) = 10.285: 0.578 | | η² | f | $\sqrt{\eta^2 / (1 - \eta^2)}$ | 0.578: f = 1.171 | | R² | f² | $R^2 / (1 - R^2)$ | 0.579: f² = 1.376 | | χ² (2 x 2) | φ | $\sqrt{\chi^2 / N}$ | 5.698, N = 80: 0.267 | | OR | d (logistic approximation) | $d = \ln(OR) \sqrt{3} / \pi$ | OR = 3.27: d = 0.654 | Two warnings. First, d and r measure different things: r depends on how the sample is split between groups, so the same d gives a smaller r when the groups are unequal. Second, the OR to d conversion assumes an underlying logistic distribution; treat it as a rough bridge, not an identity. ## Confidence intervals for effect sizes An effect size without an interval invites over-interpretation. The CI shows the range of population effects that are compatible with your data at the 95% level. Common methods: - **Noncentral t inversion for d.** The observed t follows a noncentral t distribution whose noncentrality parameter is proportional to the true d. Find the two noncentrality values that put the observed t at the 97.5th and 2.5th percentiles, then multiply by $\sqrt{1/n_1 + 1/n_2}$. This is the method in the Example A code (Steiger and Fouladi 1997; Cumming and Finch 2001). - **Large-sample standard error for d.** $SE_d \approx \sqrt{(n_1 + n_2)/(n_1 n_2) + d^2 / (2(n_1 + n_2))}$ and CI = d ± 1.96 SE. For the Lakemoor data this gives [0.16, 2.04], close to the exact [0.14, 2.04]. It is less accurate in small samples and for large d. - **Fisher z for r.** Transform, build a symmetric interval, transform back (see the formula above). - **Log scale for OR and RR.** Build the interval for ln(OR) or ln(RR) and exponentiate. - **Noncentral F for η² and ω².** Invert the noncentral F distribution in the same way as for d. Packages such as effectsize (R) automate this; check which interval they return, because conventions differ: recent effectsize versions return a one-sided interval (upper limit fixed at 1) by default, and some authors use a two-sided 90% interval so that it matches an F test at α = 0.05. State the type you report. - **Bootstrap.** Resample the data, recompute the effect size, take percentile or BCa limits (Efron 1979). Useful when no formula exists; set and report the seed. For Hedges' g, multiply both limits of the d interval by J. A wide interval is a finding in itself: it says the study cannot pin down the size of the effect, which belongs in your limitations section. ## Interpreting magnitude: why Cohen's benchmarks are a last resort Cohen (1988) offered rough conventions (d of 0.2, 0.5, 0.8; r of .10, .30, .50; η² of .01, .06, .14; f² of .02, .15, .35) and explicitly described them as a fallback for when nothing better is known. They were based on his reading of behavioural science research in the 1960s, not on any principle. Using them as fixed thresholds causes three problems. They ignore the field: a d of 0.2 on mortality is enormous, while a d of 0.8 on a reaction-time task can be routine. They ignore the cost and benefit of acting: a cheap intervention with a small effect may be worth more than an expensive one with a large effect. And they ignore uncertainty: calling d = 1.10 "large" hides that the 95% CI runs from 0.14 to 2.04, which includes effects Cohen would call small. Better ways to interpret, in order of preference: 1. **Raw units with a practical anchor.** "5.4 exam points, about half a grade band at Lakemoor." 2. **Comparison with prior evidence.** "Larger than the median effect of revision techniques in [prior review]." 3. **A smallest effect size of interest (SESOI)** fixed before the study, then ask whether the CI excludes it. 4. **Distribution-based restatements.** With d = 1.10, about 86% of the flashcard group score above the re-reading mean (the normal cumulative probability at 1.10), assuming normal distributions with equal SDs. 5. **Cohen's labels, flagged as conventions,** only when none of the above is available (Funder and Ozer 2019). ## Worked example A (fully solved): two independent groups **Question.** At Lakemoor University (fictional), did students who revised with flashcards score higher on the statistics exam than students who re-read their notes, and by how much? **Design and choice.** Two independent groups, continuous outcome, so Hedges' g with the raw mean difference. The test is the Welch t-test (the default for two independent groups); the effect size uses the pooled SD as in the formula above. **By hand from the summary statistics.** 1. Mean difference: 74.9 - 69.5 = 5.4 points. 2. Pooled SD: $s_p = \sqrt{(9 \cdot 5.466^2 + 9 \cdot 4.249^2)/18} = \sqrt{(268.9 + 162.5)/18} = \sqrt{23.97} = 4.896$. 3. Cohen's d: 5.4 / 4.896 = 1.103. 4. Correction: J = 1 - 3/(4 · 20 - 9) = 1 - 3/71 = 0.958, so g = 0.958 · 1.103 = 1.056. 5. CI for d by noncentral t inversion: [0.14, 2.04]; multiply by J for g: [0.14, 1.95]. **R code** ```r flash <- c(72, 75, 81, 66, 79, 70, 74, 84, 77, 71) reread <- c(65, 70, 74, 68, 72, 63, 77, 71, 66, 69) n1 <- length(flash); n2 <- length(reread) sp <- sqrt(((n1 - 1) * var(flash) + (n2 - 1) * var(reread)) / (n1 + n2 - 2)) d <- (mean(flash) - mean(reread)) / sp J <- 1 - 3 / (4 * (n1 + n2) - 9) # small-sample correction g <- J * d # CI for d by inverting the noncentral t distribution t_obs <- unname(t.test(flash, reread, var.equal = TRUE)$statistic) df <- n1 + n2 - 2; k <- sqrt(1 / n1 + 1 / n2) ncp_at <- function(prob) suppressWarnings( uniroot(function(ncp) pt(t_obs, df, ncp) - prob, c(-20, 20))$root) ci <- c(ncp_at(0.975), ncp_at(0.025)) * k welch <- t.test(flash, reread) # Welch is the R default round(c(sp = sp, d = d, J = J, g = g, t_pooled = t_obs, ci_low = ci[1], ci_high = ci[2], ci_g_low = J * ci[1], ci_g_high = J * ci[2]), 3) round(c(t_welch = unname(welch$statistic), df_welch = unname(welch$parameter), p_welch = welch$p.value), 3) ``` ```text sp d J g t_pooled ci_low ci_high ci_g_low 4.896 1.103 0.958 1.056 2.466 0.143 2.037 0.137 ci_g_high 1.951 t_welch df_welch p_welch 2.466 16.968 0.025 ``` **Python code** ```python import numpy as np from scipy import stats, optimize flash = np.array([72, 75, 81, 66, 79, 70, 74, 84, 77, 71]) reread = np.array([65, 70, 74, 68, 72, 63, 77, 71, 66, 69]) n1, n2 = len(flash), len(reread) sp = np.sqrt(((n1 - 1) * flash.var(ddof=1) + (n2 - 1) * reread.var(ddof=1)) / (n1 + n2 - 2)) d = (flash.mean() - reread.mean()) / sp J = 1 - 3 / (4 * (n1 + n2) - 9) t_obs = stats.ttest_ind(flash, reread).statistic # scipy default: pooled df, k = n1 + n2 - 2, np.sqrt(1 / n1 + 1 / n2) ncp_at = lambda prob: optimize.brentq( lambda nc: stats.nct.cdf(t_obs, df, nc) - prob, -20, 20) lo, hi = ncp_at(0.975) * k, ncp_at(0.025) * k welch = stats.ttest_ind(flash, reread, equal_var=False) print(f"sp={sp:.3f} d={d:.3f} J={J:.3f} g={J * d:.3f} t_pooled={t_obs:.3f}") print(f"95% CI d [{lo:.2f}, {hi:.2f}] g [{J * lo:.2f}, {J * hi:.2f}]") print(f"Welch t={welch.statistic:.3f} df={welch.df:.2f} p={welch.pvalue:.3f}") ``` ```text sp=4.896 d=1.103 J=0.958 g=1.056 t_pooled=2.466 95% CI d [0.14, 2.04] g [0.14, 1.95] Welch t=2.466 df=16.97 p=0.025 ``` **R and Python defaults differ.** `t.test()` in R runs the Welch test unless you pass `var.equal = TRUE`; `scipy.stats.ttest_ind()` runs the pooled test unless you pass `equal_var=False`. With equal group sizes the two t values are identical (2.466 here) but the df differ (18 pooled, 16.97 Welch), so the p-values differ slightly. Always state which test you ran. **Interpretation.** Flashcard students scored 5.4 points higher on average. Standardized, that is g = 1.06, but the 95% CI [0.14, 1.95] is wide: the data are compatible with anything from a small to a very large advantage. With only 10 students per group, the study shows the direction convincingly but not the size. **Report sentence.** Students who revised with flashcards scored higher (*M* = 74.9, *SD* = 5.47) than students who re-read their notes (*M* = 69.5, *SD* = 4.25), a difference of 5.4 points, Welch *t*(16.97) = 2.47, *p* = .025, Hedges' *g* = 1.06, 95% CI [0.14, 1.95]. ## Worked example B (half solved): paired design **Question.** Eight Lakemoor students took a test anxiety questionnaire before and after a study skills workshop. How large was the reduction? | Student | Before | After | Difference (before - after) | |---|---|---|---| | 1 | 24 | 21 | 3 | | 2 | 30 | 26 | 4 | | 3 | 27 | 25 | 2 | | 4 | 22 | 22 | 0 | | 5 | 35 | 30 | 5 | | 6 | 29 | 24 | 5 | | 7 | 26 | 25 | 1 | | 8 | 31 | 27 | 4 | Steps: 1. Mean difference $\bar{D}$ = 24 / 8 = [ ]. 2. SD of the differences $s_D$ = 1.852 (given). 3. d_z = $\bar{D}$ / $s_D$ = [ ]. 4. SD before = 4.140, SD after = 2.828, so the average SD = [ ]. 5. d_av = $\bar{D}$ / average SD = [ ]. 6. Check: paired t = d_z · √8 = [ ] (the software gives 4.583). 7. The correlation between before and after scores is 0.927. In one sentence, explain why d_z is about twice d_av: [ ]. 8. Which value goes into a power analysis for a repeat of this paired study, and which one would you use to compare with a between-groups study of the same workshop? [ ] ## Exercise C (unsolved): a 2 x 2 table Lakemoor recorded whether 80 students attended optional revision sessions and whether they passed the exam. | Attended | Passed | Failed | Total | |---|---|---|---| | Yes | 32 | 8 | 40 | | No | 22 | 18 | 40 | | Total | 54 | 26 | 80 | Compute: (a) the risk ratio of passing with its 95% CI; (b) the odds ratio with its 95% CI (log method); (c) φ; (d) one sentence saying which of RR and OR you would put in a report for students and why. The Pearson χ²(1) without continuity correction is 5.698. ## Code for every family The scripts below compute the effect sizes for Example B, the bakery ANOVA, the study hours correlation, Exercise C and the regression. Both print the same numbers. ```r # 1. Paired: anxiety score before and after a workshop (8 students) pre <- c(24, 30, 27, 22, 35, 29, 26, 31) post <- c(21, 26, 25, 22, 30, 24, 25, 27) dif <- pre - post d_z <- mean(dif) / sd(dif) d_av <- mean(dif) / ((sd(pre) + sd(post)) / 2) # 2. One-way ANOVA: Brightwell Bakery loaf height (cm) by yeast strain height <- c(9.8, 10.4, 10.1, 9.6, 10.0, 10.3, 10.6, 11.0, 10.8, 10.5, 11.2, 10.7, 10.2, 10.9, 10.4, 10.0, 10.6, 10.5) yeast <- factor(rep(c("A", "B", "C"), each = 6)) tab <- summary(aov(height ~ yeast))[[1]] ssb <- tab$`Sum Sq`[1]; ssw <- tab$`Sum Sq`[2] dfb <- tab$Df[1]; dfw <- tab$Df[2]; msw <- tab$`Mean Sq`[2]; Fv <- tab$`F value`[1] eta2 <- ssb / (ssb + ssw) omega2 <- (ssb - dfb * msw) / (ssb + ssw + msw) peta2 <- Fv * dfb / (Fv * dfb + dfw) # from F and df only # 3. Correlation: hours studied and exam score (12 students) hours <- c(2, 4, 5, 3, 8, 6, 7, 1, 9, 5, 6, 4) score <- c(62, 60, 74, 66, 72, 64, 80, 58, 77, 61, 70, 69) ct <- cor.test(hours, score) # Fisher z CI # 4. 2 x 2 table: attended revision sessions x passed (80 students) pass <- matrix(c(32, 8, 22, 18), nrow = 2, byrow = TRUE) chi <- chisq.test(pass, correct = FALSE) V <- sqrt(unname(chi$statistic) / (sum(pass) * (min(dim(pass)) - 1))) OR <- (32 * 18) / (8 * 22); se_lor <- sqrt(sum(1 / pass)) RR <- (32 / 40) / (22 / 40); se_lrr <- sqrt(1/32 - 1/40 + 1/22 - 1/40) # 5. Regression: score on hours and sleep sleep <- c(6, 7, 6, 8, 7, 5, 8, 6, 7, 7, 8, 5) full <- summary(lm(score ~ hours + sleep)); red <- summary(lm(score ~ hours)) f2_model <- full$r.squared / (1 - full$r.squared) f2_sleep <- (full$r.squared - red$r.squared) / (1 - full$r.squared) round(c(d_z = d_z, d_av = d_av, r_pre_post = cor(pre, post), eta2 = eta2, omega2 = omega2, partial_eta2 = peta2, F = Fv, r = unname(ct$estimate), r_low = ct$conf.int[1], r_high = ct$conf.int[2], chi2 = unname(chi$statistic), p_chi2 = chi$p.value, V = V, OR = OR, OR_low = exp(log(OR) - 1.96 * se_lor), OR_high = exp(log(OR) + 1.96 * se_lor), RR = RR, RR_low = exp(log(RR) - 1.96 * se_lrr), RR_high = exp(log(RR) + 1.96 * se_lrr), R2 = full$r.squared, adj_R2 = full$adj.r.squared, f2_model = f2_model, f2_sleep = f2_sleep), 3) ``` ```text d_z d_av r_pre_post eta2 omega2 partial_eta2 1.620 0.861 0.927 0.578 0.508 0.578 F r r_low r_high chi2 p_chi2 10.285 0.753 0.314 0.926 5.698 0.017 V OR OR_low OR_high RR RR_low 0.267 3.273 1.211 8.844 1.455 1.056 RR_high R2 adj_R2 f2_model f2_sleep 2.004 0.579 0.486 1.376 0.030 ``` ```python import numpy as np import pandas as pd from scipy import stats import statsmodels.formula.api as smf from statsmodels.stats.anova import anova_lm # 1. Paired pre = np.array([24, 30, 27, 22, 35, 29, 26, 31]) post = np.array([21, 26, 25, 22, 30, 24, 25, 27]) dif = pre - post d_z = dif.mean() / dif.std(ddof=1) d_av = dif.mean() / ((pre.std(ddof=1) + post.std(ddof=1)) / 2) # 2. One-way ANOVA bake = pd.DataFrame({ "height": [9.8, 10.4, 10.1, 9.6, 10.0, 10.3, 10.6, 11.0, 10.8, 10.5, 11.2, 10.7, 10.2, 10.9, 10.4, 10.0, 10.6, 10.5], "yeast": ["A"] * 6 + ["B"] * 6 + ["C"] * 6}) tab = anova_lm(smf.ols("height ~ C(yeast)", data=bake).fit()) ssb, ssw = tab["sum_sq"].iloc[:2] dfb, msw = tab["df"].iloc[0], tab["mean_sq"].iloc[1] eta2 = ssb / (ssb + ssw) omega2 = (ssb - dfb * msw) / (ssb + ssw + msw) # 3. Correlation with Fisher z CI hours = np.array([2, 4, 5, 3, 8, 6, 7, 1, 9, 5, 6, 4]) score = np.array([62, 60, 74, 66, 72, 64, 80, 58, 77, 61, 70, 69]) res_r = stats.pearsonr(hours, score) r_ci = res_r.confidence_interval() # Fisher z by default # 4. 2 x 2 table table = np.array([[32, 8], [22, 18]]) chi2, p, dof, _ = stats.chi2_contingency(table, correction=False) V = np.sqrt(chi2 / (table.sum() * (min(table.shape) - 1))) OR = stats.contingency.odds_ratio(table, kind="sample").statistic or_ci = np.exp(np.log(OR) + np.array([-1.96, 1.96]) * np.sqrt((1 / table).sum())) rr = stats.contingency.relative_risk(32, 40, 22, 40) rr_ci = rr.confidence_interval() # 5. Regression dat = pd.DataFrame({"score": score, "hours": hours, "sleep": [6, 7, 6, 8, 7, 5, 8, 6, 7, 7, 8, 5]}) full = smf.ols("score ~ hours + sleep", data=dat).fit() red = smf.ols("score ~ hours", data=dat).fit() print(f"d_z={d_z:.3f} d_av={d_av:.3f}") print(f"eta2={eta2:.3f} omega2={omega2:.3f}") print(f"r={res_r.statistic:.3f} [{r_ci.low:.3f}, {r_ci.high:.3f}]") print(f"chi2={chi2:.3f} p={p:.3f} V={V:.3f} OR={OR:.2f} [{or_ci[0]:.2f}, {or_ci[1]:.2f}]" f" RR={rr.relative_risk:.2f} [{rr_ci.low:.2f}, {rr_ci.high:.2f}]") print(f"R2={full.rsquared:.3f} f2_model={full.rsquared / (1 - full.rsquared):.3f} " f"f2_sleep={(full.rsquared - red.rsquared) / (1 - full.rsquared):.3f}") ``` ```text d_z=1.620 d_av=0.861 eta2=0.578 omega2=0.508 r=0.753 [0.314, 0.926] chi2=5.698 p=0.017 V=0.267 OR=3.27 [1.21, 8.84] RR=1.45 [1.06, 2.00] R2=0.579 f2_model=1.376 f2_sleep=0.030 ``` Notice three teaching points in this output. The bakery ω² (0.508) is smaller than η² (0.578), as it always is. The regression explains 57.9% of the variance overall, yet adding sleep to a model that already has hours adds almost nothing (f² = 0.030): the model-level effect size and the effect size of one predictor answer different questions. And `chisq.test()` in R applies a continuity correction to 2 x 2 tables by default, as does `scipy.stats.chi2_contingency()`; both scripts switch it off with `correct = FALSE` and `correction=False` so that V matches the uncorrected χ². ## Reading software output | What you see | Where | What it means | What to report | |---|---|---|---| | `t`, `df` from `t.test()` or `ttest_ind()` | Test line | Test statistic; check whether it is Welch or pooled | t, df, p; then d or g from the means and SDs | | `Sum Sq`, `Mean Sq`, `F value` (R) or `sum_sq`, `mean_sq`, `F` (statsmodels) | ANOVA table | Ingredients of η², partial η², ω² | F(df1, df2), p, ω² or η² | | `cor` and `95 percent confidence interval` | `cor.test()` | r is the effect size; CI by Fisher z | r, 95% CI, n | | `X-squared` | `chisq.test()` | χ² with or without continuity correction | χ²(df, N = ...), p, V or φ, OR or RR with CI | | `Multiple R-squared`, `Adjusted R-squared` | `summary(lm())` | Variance explained by the model | R², adjusted R², F test; f² for a key predictor set | | `Estimate` on the log-odds scale | `glm(..., family = binomial)` | Log odds ratio | exp(Estimate) as OR with exp of the CI limits | ## Reporting template and checklist Fill-in sentence: > [Group 1] scored [higher / lower] (*M* = [ ], *SD* = [ ]) than [group 2] (*M* = [ ], *SD* = [ ]), a difference of > [raw difference and units], [test] *t*([df]) = [ ], *p* = [ ], [Hedges' *g* / *d_z* / *d_av*] = [ ], 95% CI [ , ]. | Analysis | Effect size reported | Estimate | 95% CI | CI method | Interpretation anchor | |---|---|---|---|---|---| | *Example:* Flashcards vs re-reading | Hedges' g | 1.06 | [0.14, 1.95] | Noncentral t | 5.4 exam points | | *Example:* Revision sessions x pass | Risk ratio | 1.45 | [1.06, 2.00] | Log method | 80% vs 55% pass rate | | [Analysis] | [ ] | [ ] | [ , ] | [ ] | [ ] | Checklist: - [ ] The effect size matches the design (independent, paired, factorial, categorical). - [ ] For paired designs, the text says whether d is d_z or d_av. - [ ] The raw (unstandardized) effect is reported when the units have meaning. - [ ] Every effect size has a 95% CI and the CI method is stated. - [ ] Hedges' g (not d) is used for small samples, or the choice is explained. - [ ] Partial η² is labelled as partial; ω² is preferred when samples are small. - [ ] Interpretation compares with a practical anchor or prior evidence before any Cohen label. - [ ] R and Python (or two tools) give the same numbers, or the difference is explained. ## Common mistakes and misconceptions | Mistake | Why it is wrong | Fix | |---|---|---| | "p < .001, so the effect is large" | p depends on n; a tiny effect gives a tiny p with enough data | Report and interpret the effect size and its CI separately | | Reporting "d" for a paired design without saying which | d_z and d_av can differ by a factor of two (1.62 vs 0.86 in Example B) | Name the version and the denominator | | Converting a paired t with the independent formula | $t\sqrt{1/n_1 + 1/n_2}$ assumes two groups | Use $d_z = t/\sqrt{n}$ | | Adding partial η² values across effects | Each has a different denominator, so they can sum to more than 1 | Report each separately, or use η² when you need shares that add up | | Calling d = 0.5 "medium" in every field | Cohen's benchmarks are field-free conventions from 1988 | Use raw units, prior studies or a SESOI | | Treating OR as RR | When the outcome is common (pass rate 55% to 80%), OR (3.27) is far larger than RR (1.45) | Report RR for common outcomes or explain the OR | | Effect size without CI | A point estimate from n = 20 is very uncertain | Always add the 95% CI | | Using η² from a small sample as the population value in power analysis | η² overestimates; the study will be underpowered | Use ω², or a conservative value, for planning | ## Self-test questions 1. A study reports *t*(38) = 2.10 for two independent groups of 20. What is Cohen's d? 2. Why is Hedges' g always slightly smaller in absolute value than Cohen's d, and when does the difference stop mattering? 3. A one-way ANOVA has η² = 0.12. Is partial η² larger, smaller or the same? Is ω² larger, smaller or the same? 4. In a 2 x 2 table the pass rates are 90% and 75%. Without computing, will the odds ratio be closer to or further from 1 than the risk ratio? 5. A colleague writes "R² = .58, a large effect". What two things would you ask them to add? 6. Which effect size do you put into a power analysis for a paired t-test, d_z or d_av, and why? ## My notes and next steps | Item | Your entry | |---|---| | Confidence with choosing an effect size (1 to 5) | [ ] | | Confidence with computing CIs for effect sizes (1 to 5) | [ ] | | Effect sizes I still confuse | [e.g. partial η² and η²] | | Open questions for my tutor or supervisor | [ ] | | Errors logged in stats-error-log | [Link or IDs] | | Next template to study | [stats-power-sample-size] | | Next review date | YYYY-MM-DD | ## Sources and further reading Based on the following sources, cited by name; all text, notation and examples here are original. - Cohen, J. (1988). *Statistical Power Analysis for the Behavioral Sciences* (2nd ed.). Lawrence Erlbaum Associates. - Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. *Journal of Educational Statistics*, 6(2), 107–128. - Hedges, L. V., and Olkin, I. (1985). *Statistical Methods for Meta-Analysis*. Academic Press. - Olejnik, S., and Algina, J. (2003). Generalized eta and omega squared statistics: measures of effect size for some common research designs. *Psychological Methods*, 8(4), 434–447. - Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. *Frontiers in Psychology*, 4, 863. - Borenstein, M., Hedges, L. V., Higgins, J. P. T., and Rothstein, H. R. (2009). *Introduction to Meta-Analysis*. Wiley. - Steiger, J. H., and Fouladi, R. T. (1997). Noncentrality interval estimation and the evaluation of statistical models. In L. L. Harlow, S. A. Mulaik, and J. H. Steiger (Eds.), *What If There Were No Significance Tests?* Lawrence Erlbaum Associates. - Cumming, G., and Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. *Educational and Psychological Measurement*, 61(4), 532–574. - Fisher, R. A. (1921). On the "probable error" of a coefficient of correlation deduced from a small sample. *Metron*, 1(4), 3–32. - Efron, B. (1979). Bootstrap methods: another look at the jackknife. *The Annals of Statistics*, 7(1), 1–26. - Funder, D. C., and Ozer, D. J. (2019). Evaluating effect size in psychological research: sense and nonsense. *Advances in Methods and Practices in Psychological Science*, 2(2), 156–168. - Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. *The American Statistician*, 70(2), 129–133. - Further reading (by name only): NIST/SEMATECH e-Handbook of Statistical Methods (public domain); de Smith, M. J. (2024). *Statistical Analysis Handbook* (2024 edition, StatsRef); *OpenIntro Statistics*. ## Answer key **Example B.** (1) $\bar{D}$ = 3.0. (3) d_z = 3.0 / 1.852 = 1.620. (4) Average SD = (4.140 + 2.828) / 2 = 3.484. (5) d_av = 3.0 / 3.484 = 0.861. (6) 1.620 · √8 = 1.620 · 2.828 = 4.58. (7) Because before and after scores are highly correlated (r = 0.927), the SD of the differences (1.852) is much smaller than the raw SDs, so dividing by it gives a larger number. (8) d_z goes into the paired power analysis because it is what the paired t-test detects; d_av is the one to compare with a between-groups study. **Exercise C.** (a) RR = (32/40) / (22/40) = 0.80 / 0.55 = 1.45; SE of ln RR = √(1/32 - 1/40 + 1/22 - 1/40) = 0.163; 95% CI [1.06, 2.00]. (b) OR = (32 · 18) / (8 · 22) = 576 / 176 = 3.27; SE of ln OR = √(1/32 + 1/8 + 1/22 + 1/18) = 0.507; 95% CI [1.21, 8.84]. (c) φ = (32 · 18 - 8 · 22) / √(40 · 40 · 54 · 26) = 400 / 1498.8 = 0.267, which equals √(5.698 / 80). (d) The risk ratio: "attenders were 1.45 times as likely to pass (80% vs 55%)" is easy to understand, while the OR of 3.27 would be misread as "three times as likely" because passing is common. **Self-test.** 1. d = 2.10 · √(1/20 + 1/20) = 2.10 · 0.316 = 0.66. 2. J is below 1 because d overestimates the population effect in small samples. The shrinkage is about 3/(4 df), so it is about 4% at df = 18 and under 2% once df is above about 50, where it rarely changes the interpretation. 3. The same: in a one-way design there is only one effect, so partial η² = η². ω² is smaller. 4. Further from 1. The OR exaggerates relative to the RR when the outcome is common: here RR = 0.90/0.75 = 1.20 and OR = (0.90/0.10) / (0.75/0.25) = 3.0. 5. A confidence interval (or at least adjusted R² and n), and an anchor for "large": compared with what prior models or practical need? Also whether any single predictor matters (f² for the predictor of interest). 6. d_z, because the paired t statistic is d_z · √n. Using d_av would misstate the power: it underestimates power when the before-after correlation is above 0.5 (as here, r = 0.927, where d_z > d_av) and overestimates it when the correlation is below 0.5. ## Revision history | Version | Date | Author | Change summary | |---|---|---|---| | 0.1 | YYYY-MM-DD | [Your name] | First draft from template | | [0.2] | YYYY-MM-DD | [Your name] | *Example:* Replaced Lakemoor data with thesis data; reran R and Python |