# Power Analysis and Sample Size: [Your study question or course] > A worksheet for planning how many participants, units or observations a study needs: the inputs you must decide > (α, power, effect size, design, allocation ratio, attrition), analytic power calculations for t-tests, two > proportions, one-way ANOVA, correlation and multiple regression in R and Python, simulation-based power for designs > without a formula, a sensitivity analysis table, and a reporting sentence. It is for statistics students learning > what power means, master's and PhD students who must justify a sample size to a committee or ethics board, and > analysts sizing an experiment. It teaches by structure: decide the inputs in order, compute, check with a second > tool, then test yourself. The worked examples use fictional data; replace them with your own. > Source basis: Cohen (1988) for the effect size conventions and power framework; Lakens (2022) for sample size > justification; all text and examples are original. ## Document control This table records the study the calculation belongs to and the software that produced each number. Rows marked *Example:* show how to fill a field. | Field | Value | |---|---| | Course or project | [STAT 301 Research Methods / thesis study 2] | | Learner or analyst | [Your name] | | Data set | [Pilot data or prior study used for the effect size] *Example:* Lakemoor revision pilot (fictional), 20 students | | Software and versions | R 4.5.1 with pwr 1.3.0; Python 3.14, numpy 2.5.4, scipy 1.18.1, statsmodels 0.15.0 | | Date | YYYY-MM-DD | | Version | [0.1] | ## What power is and why it matters **Statistical power** is the probability that a study will detect an effect of a given size, if that effect really exists. "Detect" means the test gives p below α. Power depends on four quantities, and if you fix any three, the fourth is determined: the significance level α, the effect size, the sample size n, and the power itself (1 - β, where β is the Type II error rate, the chance of missing a real effect). A sample size calculation fixes α, power and the effect size and solves for n. A **sensitivity analysis** fixes α, power and n and solves for the smallest effect the study can detect. Why it matters. An underpowered study is not just "less certain". It wastes participants' time, it usually ends in a non-significant result that is hard to interpret, and when it does reach significance the estimate is likely to be exaggerated, because only the luckiest overestimates cross the threshold. An overpowered study spends resources that could have gone elsewhere and can make trivially small effects significant. Planning power is how you match the study to the question. Power is a property of the design and the assumed effect, not of the data you later collect. Computing "observed power" from your own results after the study adds nothing: it is a direct function of the p-value you already have (Hoenig and Heisey 2001). ## The planning workflow The diagram shows the order of decisions. The design determines the test, the test determines which effect size the software needs, and the effect size must be justified before any calculation. ```mermaid flowchart TD A["State the primary hypothesis and the one primary outcome"] --> B["Choose the design and the test"] B --> C["Set alpha (usually 0.05, two-sided)"] C --> D["Set target power (0.80 or 0.90)"] D --> E["Justify the effect size"] E --> E1["Smallest effect of interest"] E --> E2["Prior literature (bias-adjusted)"] E --> E3["Pilot data (use with caution)"] E1 --> F["Is there a formula for this design?"] E2 --> F E3 --> F F -->|Yes| G["Analytic power: pwr or statsmodels"] F -->|No| H["Simulation with a fixed seed"] G --> I["Round up, add attrition, round up again"] H --> I I --> J["Sensitivity table for the feasible n"] J --> K["Write the justification sentence"] ``` ## Input worksheet | Input | Value | Reason or source | |---|---|---| | Primary outcome | [ ] *Example:* Exam score (0 to 100) | Pre-registered primary outcome | | Design and test | [ ] *Example:* Two independent groups, Welch t-test | Students randomised to revision method | | α (Type I error rate) | [0.05] | Field convention; two-sided | | Sidedness | [Two-sided] | Default; one-sided only with a written, prior reason | | Target power (1 - β) | [0.80 / 0.90] | 0.90 when a missed effect is costly | | Effect size (in the metric the software needs) | [ ] *Example:* d = 0.5 | SESOI: 5 points; SD 10 from past cohorts' records | | Allocation ratio n2 : n1 | [1 : 1] | Equal allocation is most efficient for a fixed total | | Expected attrition | [ ] *Example:* 15% | Previous Lakemoor course studies | | Number of primary tests (multiplicity) | [1] | If more, adjust α (Bonferroni or Holm) | | Resulting n per group (before attrition) | [ ] | From the code below | | n to recruit per group (after attrition) | [ ] | n / (1 - attrition), rounded up | ## The formulas behind the software The idea is simple. Under the null hypothesis the test statistic is centred at zero; under the alternative it is shifted by an amount that grows with the effect size and with the square root of n. Power is the share of the shifted distribution that lies beyond the critical value. For two independent groups of equal size n, the normal approximation gives: $$ n \approx \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^2}{d^2} $$ With α = 0.05 (z = 1.960) and power 0.80 (z = 0.842), $(1.960 + 0.842)^2 = 7.85$, so $n \approx 15.7 / d^2$: for d = 0.5 that is 62.8. The exact calculation uses the noncentral t distribution and gives 63.77, so 64 per group. Halving the effect size quadruples n: that single fact explains why small effects need very large studies. The exact power for the two-sample t-test is the probability that a noncentral t variable exceeds the critical value: $$ 1 - \beta = P\left(|T_{\nu,\lambda}| > t_{1-\alpha/2,\,\nu}\right), \qquad \nu = 2n - 2, \qquad \lambda = d\sqrt{n/2} $$ The other designs use the same idea with a different effect size metric: | Design | Effect size the software needs | Conversion | |---|---|---| | Two independent means | Cohen's d | d = (μ1 - μ2) / σ | | Paired means | d_z (mean difference / SD of differences) | $d_z = d_{av}/\sqrt{2(1-\rho)}$ if the SDs are equal | | Two proportions | Cohen's h | $h = 2\arcsin\sqrt{p_1} - 2\arcsin\sqrt{p_2}$ | | One-way ANOVA, k groups | Cohen's f | $f = \sigma_{\text{means}} / \sigma = \sqrt{\eta^2/(1-\eta^2)}$ | | Correlation | r | Fisher z: $n \approx \left(\frac{z_{1-\alpha/2} + z_{1-\beta}}{\tanh^{-1} r}\right)^2 + 3$ | | Multiple regression, u predictors tested | f² | $f^2 = R^2/(1-R^2)$ or the change in R² over the reduced model | Attrition and allocation adjustments: $$ n_{\text{recruit}} = \left\lceil \frac{n}{1 - a} \right\rceil, \qquad \text{allocation } k = n_2/n_1: \; n_1 \approx \frac{(1 + 1/k)\,(z_{1-\alpha/2} + z_{1-\beta})^2}{d^2} $$ | Symbol | Meaning | |---|---| | $\alpha$ | Significance level (Type I error rate), 0.05 unless stated | | $\beta$ | Type II error rate; power = 1 - β | | $z_q$ | Standard normal quantile at probability q | | $d, h, f, f^2, r$ | Effect size metrics (see table) | | $\nu, \lambda$ | Degrees of freedom df and noncentrality parameter | | $a$ | Expected attrition proportion | | $k$ | Allocation ratio n2 / n1 | ## Analytic power in R and Python The scripts below solve for n in six standard designs at α = 0.05 (two-sided) and power 0.80: two independent groups (d = 0.5), paired (d_z = 0.4), two proportions (60% vs 45%), one-way ANOVA with three groups (f = 0.25), correlation (r = 0.30) and multiple regression with three predictors (f² = 0.15). The R script also checks a 2:1 allocation and prints a sensitivity table. ```r library(pwr) # install.packages("pwr") once t_two <- pwr.t.test(d = 0.5, sig.level = 0.05, power = 0.80, type = "two.sample")$n t_paired <- pwr.t.test(d = 0.4, power = 0.80, type = "paired")$n h <- ES.h(0.60, 0.45) # Cohen's h for two proportions two_prop <- pwr.2p.test(h = h, power = 0.80)$n anova_n <- pwr.anova.test(k = 3, f = 0.25, power = 0.80)$n # per group corr_n <- pwr.r.test(r = 0.30, power = 0.80)$n reg_v <- pwr.f2.test(u = 3, f2 = 0.15, power = 0.80)$v # denominator df round(c(t_two = t_two, t_paired = t_paired, h = h, two_prop = two_prop, anova_per_group = anova_n, corr = corr_n, reg_v = reg_v, reg_N = reg_v + 3 + 1), 2) # unequal allocation 2:1, check the power of n1 = 48, n2 = 96 round(pwr.t2n.test(n1 = 48, n2 = 96, d = 0.5)$power, 3) # base R cross-checks (no package) round(c(power.t.test(delta = 5, sd = 10, power = 0.80)$n, power.prop.test(p1 = 0.60, p2 = 0.45, power = 0.80)$n), 2) # sensitivity: n = 30 per group round(pwr.t.test(n = 30, power = 0.80)$d, 3) round(sapply(c(0.3, 0.5, 0.7, 0.9), function(d) pwr.t.test(n = 30, d = d)$power), 3) # required n per group for a grid of d and power grid <- outer(c(0.3, 0.4, 0.5, 0.6, 0.8), c(0.80, 0.90), Vectorize(function(d, p) ceiling(pwr.t.test(d = d, power = p)$n))) dimnames(grid) <- list(d = c(0.3, 0.4, 0.5, 0.6, 0.8), power = c("0.80", "0.90")); grid ``` ```text t_two t_paired h two_prop anova_per_group 63.77 51.01 0.30 172.66 52.40 corr reg_v reg_N 84.07 72.71 76.71 [1] 0.802 [1] 63.77 172.80 [1] 0.736 [1] 0.208 0.478 0.760 0.929 power d 0.80 0.90 0.3 176 235 0.4 100 133 0.5 64 86 0.6 45 60 0.8 26 34 ``` ```python import numpy as np from scipy import stats from statsmodels.stats.power import (TTestIndPower, TTestPower, NormalIndPower, FTestAnovaPower, FTestPowerF2) from statsmodels.stats.proportion import proportion_effectsize t_two = TTestIndPower().solve_power(effect_size=0.5, alpha=0.05, power=0.80) t_ratio = TTestIndPower().solve_power(effect_size=0.5, alpha=0.05, power=0.80, ratio=2) t_paired = TTestPower().solve_power(effect_size=0.4, alpha=0.05, power=0.80) h = proportion_effectsize(0.60, 0.45) two_prop = NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.80) anova_total = FTestAnovaPower().solve_power(effect_size=0.25, k_groups=3, alpha=0.05, power=0.80) reg = FTestPowerF2().solve_power(effect_size=0.15, df_num=3, df_denom=None, alpha=0.05, power=0.80) # correlation: Fisher z approximation (statsmodels has no correlation power class) z_a, z_b = stats.norm.ppf(0.975), stats.norm.ppf(0.80) corr = ((z_a + z_b) / np.arctanh(0.30)) ** 2 + 3 print(f"t two-sample n/group={t_two:.2f}; ratio 2:1 n1={t_ratio:.2f} n2={2 * t_ratio:.2f}") print(f"paired n={t_paired:.2f}; h={h:.3f}; two proportions n/group={two_prop:.2f}") print(f"ANOVA total N={anova_total:.2f} ({anova_total / 3:.2f} per group)") print(f"regression df_denom={reg:.2f} -> N={reg + 3 + 1:.2f}; correlation n={corr:.2f}") print(f"sensitivity d (n=30/group)={TTestIndPower().solve_power(nobs1=30, alpha=0.05, power=0.80):.3f}") ``` ```text t two-sample n/group=63.77; ratio 2:1 n1=47.74 n2=95.48 paired n=51.01; h=0.302; two proportions n/group=172.66 ANOVA total N=157.19 (52.40 per group) regression df_denom=72.71 -> N=76.71; correlation n=84.93 sensitivity d (n=30/group)=0.736 ``` Results after rounding up: | Design | Effect size | n (R pwr) | n (Python statsmodels) | Recruit | |---|---|---|---|---| | Two independent groups | d = 0.5 | 63.77 per group | 63.77 per group | 64 per group, 128 total | | Two groups, 2:1 allocation | d = 0.5 | n1 = 48, n2 = 96 gives power 0.802 | n1 = 47.74, n2 = 95.48 | 48 + 96 = 144 total | | Paired | d_z = 0.4 | 51.01 pairs | 51.01 pairs | 52 pairs | | Two proportions, 60% vs 45% | h = 0.302 | 172.66 per group | 172.66 per group | 173 per group | | One-way ANOVA, 3 groups | f = 0.25 | 52.40 per group | 157.19 total | 53 per group, 159 total | | Correlation | r = 0.30 | 84.07 | 84.93 | 85 | | Multiple regression, 3 predictors | f² = 0.15 | v = 72.71, N = 76.71 | df_denom = 72.71 | 77 | **Why the tools differ, and why it matters.** - **ANOVA n per group vs total.** `pwr.anova.test()` returns n per group; `FTestAnovaPower` returns the total number of observations. 52.40 · 3 = 157.19. Mixing the two up gives a study three times too large or too small. - **Correlation.** `pwr.r.test()` uses a refined approximation and gives 84.07; the plain Fisher z formula in the Python code gives 84.93. Both round up to 85, but in general state which method you used. - **Two proportions.** `pwr.2p.test()` and statsmodels use Cohen's h with the arcsine transformation (172.66); base R `power.prop.test()` uses a different normal approximation (172.80). Both give 173 here. - **Regression.** Both tools return the denominator df v, not N. For u tested predictors and no others, N = v + u + 1; if the model also has covariates that are not tested, add them as well. - **Welch vs pooled.** The t-test power functions assume equal variances (pooled t). With equal group sizes and similar SDs the Welch test has nearly the same power, so the result transfers; with unequal sizes and unequal SDs, simulate. - **Unequal allocation costs participants.** A 2:1 split needs 144 people for the same power that a 1:1 split reaches with 128. Use unequal allocation only for a reason such as cost or ethics. The chart shows how power grows with n per group for three effect sizes, computed with `pwr.t.test()`. The dashed line marks 0.80. ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "description": "Power of a two-sided two-sample t-test at alpha 0.05 by n per group", "width": 380, "height": 220, "data": {"values": [ {"n": 10, "d": "d = 0.3", "power": 0.097}, {"n": 20, "d": "d = 0.3", "power": 0.152}, {"n": 30, "d": "d = 0.3", "power": 0.208}, {"n": 40, "d": "d = 0.3", "power": 0.263}, {"n": 60, "d": "d = 0.3", "power": 0.371}, {"n": 80, "d": "d = 0.3", "power": 0.471}, {"n": 100, "d": "d = 0.3", "power": 0.560}, {"n": 150, "d": "d = 0.3", "power": 0.736}, {"n": 200, "d": "d = 0.3", "power": 0.849}, {"n": 10, "d": "d = 0.5", "power": 0.185}, {"n": 20, "d": "d = 0.5", "power": 0.338}, {"n": 30, "d": "d = 0.5", "power": 0.478}, {"n": 40, "d": "d = 0.5", "power": 0.598}, {"n": 60, "d": "d = 0.5", "power": 0.775}, {"n": 80, "d": "d = 0.5", "power": 0.882}, {"n": 100, "d": "d = 0.5", "power": 0.940}, {"n": 150, "d": "d = 0.5", "power": 0.991}, {"n": 200, "d": "d = 0.5", "power": 0.999}, {"n": 10, "d": "d = 0.8", "power": 0.395}, {"n": 20, "d": "d = 0.8", "power": 0.693}, {"n": 30, "d": "d = 0.8", "power": 0.861}, {"n": 40, "d": "d = 0.8", "power": 0.942}, {"n": 60, "d": "d = 0.8", "power": 0.991}, {"n": 80, "d": "d = 0.8", "power": 0.999}, {"n": 100, "d": "d = 0.8", "power": 1.000}, {"n": 150, "d": "d = 0.8", "power": 1.000}, {"n": 200, "d": "d = 0.8", "power": 1.000} ]}, "layer": [ { "mark": {"type": "line", "point": true}, "encoding": { "x": {"field": "n", "type": "quantitative", "title": "n per group"}, "y": {"field": "power", "type": "quantitative", "title": "Power", "scale": {"domain": [0, 1]}}, "color": {"field": "d", "type": "nominal", "title": "Effect size"} } }, { "data": {"values": [{"target": 0.8}]}, "mark": {"type": "rule", "strokeDash": [4, 4]}, "encoding": {"y": {"field": "target", "type": "quantitative"}} } ] } ``` ## Simulation-based power When no formula fits your planned analysis (a rank test on skewed data, a mixed model, a count model with overdispersion, a test after data-dependent exclusions), estimate power by simulation: 1. Write down a data-generating model that matches what you expect: distributions, effect, variability. 2. Generate one fake data set of the planned size and run exactly the analysis you will run on the real data. 3. Record whether p < α. Repeat thousands of times with a fixed seed. 4. Power is the proportion of replications with p < α. Its Monte Carlo standard error is $\sqrt{\hat{p}(1-\hat{p})/R}$ for R replications; with R = 5,000 and power near 0.80, that is about 0.006. 5. Repeat over a grid of n and pick the smallest n that reaches the target. *Example:* Lakemoor wants to compare the time students take to complete an online quiz with an old and a new interface. Times are right-skewed, so the planned test is the Wilcoxon rank-sum (Mann-Whitney) test. The model: log-normal times with median 20 minutes under the old interface, a 30% shorter median under the new one (log-scale shift 0.35), and log-scale SD 0.6. ```r # Simulated power: skewed task times (minutes), Mann-Whitney / Wilcoxon rank-sum test set.seed(2026) sim_power <- function(n, reps = 5000, shift = 0.35) { p_w <- p_t <- numeric(reps) for (i in seq_len(reps)) { old <- rlnorm(n, meanlog = log(20), sdlog = 0.6) new <- rlnorm(n, meanlog = log(20) - shift, sdlog = 0.6) p_w[i] <- wilcox.test(old, new)$p.value p_t[i] <- t.test(old, new)$p.value # Welch, for comparison } c(n = n, wilcoxon = mean(p_w < 0.05), welch = mean(p_t < 0.05)) } round(t(sapply(c(30, 40, 50, 55, 60), sim_power)), 3) ``` ```text n wilcoxon welch [1,] 30 0.588 0.531 [2,] 40 0.707 0.656 [3,] 50 0.790 0.738 [4,] 55 0.837 0.797 [5,] 60 0.873 0.828 ``` ```python import numpy as np from scipy import stats rng = np.random.default_rng(2026) def sim_power(n, reps=5000, shift=0.35): p_w = np.empty(reps) p_t = np.empty(reps) for i in range(reps): old = rng.lognormal(mean=np.log(20), sigma=0.6, size=n) new = rng.lognormal(mean=np.log(20) - shift, sigma=0.6, size=n) p_w[i] = stats.mannwhitneyu(old, new).pvalue p_t[i] = stats.ttest_ind(old, new, equal_var=False).pvalue return n, (p_w < 0.05).mean(), (p_t < 0.05).mean() for n in (30, 40, 50, 55, 60): print("n=%d wilcoxon=%.3f welch=%.3f" % sim_power(n)) ``` ```text n=30 wilcoxon=0.581 welch=0.541 n=40 wilcoxon=0.711 welch=0.658 n=50 wilcoxon=0.795 welch=0.747 n=55 wilcoxon=0.835 welch=0.787 n=60 wilcoxon=0.873 welch=0.835 ``` R and Python use different random number generators, so the same seed gives different streams and the estimates differ by Monte Carlo error (0.790 vs 0.795 at n = 50). Both agree that 50 per group falls just short and 55 per group reaches about 0.84 for the rank test. They also show a design lesson: on skewed data the rank test has more power than the Welch t-test on the raw times. Report: "Power was estimated by simulation (5,000 replications, seed 2026, R 4.5.1) under a log-normal model ..." ## Justifying the effect size The effect size is the input that drives n the most and the one most often chosen badly. Three legitimate routes, in order of preference (Lakens 2022): 1. **Smallest effect size of interest (SESOI).** Ask what effect would matter in practice or for theory, then power the study to detect that. *Example:* the Lakemoor exam board considers 5 points (half a grade band) the smallest improvement worth a change in teaching; the score SD in past cohorts is about 10, so d = 0.5. A SESOI is the most defensible choice because it does not depend on guessing the true effect. 2. **Prior literature, adjusted for bias.** Published effects are inflated by publication bias and by the winner's curse. Use a meta-analytic estimate if one exists, prefer the lower part of its CI, and say how you adjusted. 3. **Pilot data, with caution.** A pilot effect size is very imprecise. The Lakemoor pilot of 10 + 10 students gave g = 1.06, 95% CI [0.14, 1.95] (see stats-effect-sizes). Planning with d = 1.10 gives 14.0, so 15 per group; if the true effect is 0.5, those 15 per group have power 0.26. Use a pilot mainly to test procedures. If you must plan on a pilot effect, use a conservative value: "safeguard power" (Perugini, Gallucci and Costantini 2014) plans on the lower limit of a two-sided 60% CI, here d = 0.68, which gives 35 per group. Pilot SDs are imprecise too (next paragraph), and when only promising pilots go ahead the follow-up study is planned on an inflated effect (Albers and Lakens 2018). **Where the SD comes from.** When the SESOI is set in raw units (5 exam points), the SD you divide by decides d, so it needs its own source. Prefer an SD from a large, similar population: past cohorts, registry or administrative data, or the control arms of earlier studies. An SD estimated from a pilot of 10 per group is itself uncertain: with 9 df, the 95% CI for the population SD runs from about 0.69 to 1.83 times the sample SD, and a small, homogeneous pilot group often understates the spread of the full population. If the only SD you have is from a pilot, plan on a larger value (for example the upper limit of its CI) and say so. Avoid "we expect a medium effect (d = 0.5) following Cohen (1988)" as the only justification. It says nothing about your study. If you use a benchmark, say why it is a sensible minimum in your field. ## Sensitivity analysis When n is fixed by circumstances, report what the study can detect. With 30 students per group, α = 0.05 two-sided, the smallest d detectable with 80% power is 0.736, and power at other effect sizes is: | True d | Power with n = 30 per group | Reading | |---|---|---| | 0.3 | 0.208 | Mostly missed | | 0.5 | 0.478 | A coin flip | | 0.7 | 0.760 | Close to target | | 0.9 | 0.929 | Reliably detected | And the reverse view, n per group needed for each combination of d and target power: | d | Power 0.80 | Power 0.90 | |---|---|---| | 0.3 | 176 | 235 | | 0.4 | 100 | 133 | | 0.5 | 64 | 86 | | 0.6 | 45 | 60 | | 0.8 | 26 | 34 | | Your study | Value | |---|---| | Feasible n per group | [ ] | | Smallest effect detectable at 80% power | [ ] | | Is that effect smaller than or equal to your SESOI? | [Yes / No: if no, say how you will interpret a null result] | ## Worked example A (fully solved): two independent groups with attrition **Question.** Lakemoor plans a full trial of flashcard revision against re-reading. How many students should be recruited? 1. **Design and test.** Two independent groups, randomised; Welch t-test on exam score; α = 0.05, two-sided. 2. **Effect size.** SESOI of 5 points (set by the exam board). Planning SD 10, taken from the exam records of the past five full cohorts, so d = 5 / 10 = 0.5 (not the pilot's g = 1.06). The pilot SDs (5.5 and 4.2) come from 10 students per group in one seminar, so they are imprecise and likely too small for a whole cohort. Had the plan used SD 5, the same 5-point SESOI would become d = 1.0 and about 17 per group; if the true SD is 10, that study would have power of only about 0.29. 3. **Power.** 0.80. 4. **Compute.** `pwr.t.test(d = 0.5, power = 0.80)` and `TTestIndPower().solve_power(effect_size=0.5, ...)` both give 63.77, so 64 per group. Base R `power.t.test(delta = 5, sd = 10)` agrees in raw units. 5. **Hand check.** Normal approximation: 2 · 7.85 / 0.25 = 62.8, a little below the exact t-based value, as expected. 6. **Attrition.** 15% of students usually miss the exam: 64 / 0.85 = 75.3, so recruit 76 per group, 152 in total. 7. **Check.** With 76 per group and d = 0.5, power is 0.865 if nobody drops out, and about 0.80 if 15% drop out. 8. **Report sentence.** "We will recruit 152 students (76 per group). With 64 students per group after an expected 15% attrition, a two-sided Welch t-test at α = .05 has 80% power to detect a difference of 5 exam points (d = 0.5), the smallest improvement the exam board considers worth a change in teaching. Calculations used the pwr package 1.3.0 in R 4.5.1 and were checked with statsmodels 0.15.0." ## Worked example B (half solved): two proportions **Question.** Lakemoor wants to know whether a weekly reminder email raises the pass rate in an introductory course from 45% to 60%. 1. Design and test: two independent groups, binary outcome, test of two proportions; α = 0.05, two-sided, power 0.80. 2. Effect size metric needed by `pwr.2p.test()`: Cohen's [ ]. 3. h = 2 arcsin √0.60 - 2 arcsin √0.45 = [ ] (to three decimals). 4. n per group from the code output: 172.66, so [ ] per group. 5. Expected attrition 10%: n to recruit per group = [ ] / 0.90 = [ ], rounded up to [ ]; total [ ]. 6. Base R `power.prop.test()` gives 172.80. Why does it differ slightly from `pwr.2p.test()`? [ ] 7. Write the report sentence: [ ]. ## Exercise C (unsolved) (a) A Harbour City Library analyst wants to detect a correlation of r = 0.25 between weekly opening hours and visits across branch-weeks, with α = 0.05 two-sided and power 0.90. How many observations are needed? (b) A Brightwell Bakery quality study will regress loaf volume on four process variables (proofing time, oven temperature, hydration, flour protein). The team wants 80% power to detect f² = 0.10 for the set of four predictors. How many loaves must be measured? ## Reporting template and checklist Fill-in sentence: > A sample size of [N total] ([n per group] per group, [allocation]) was planned to give [power]% power to detect > [effect size and metric, with raw units] using a [two-sided / one-sided] [test] at α = [ ]. The effect size was > based on [SESOI / meta-analysis / pilot, with reason]. The target was inflated by [attrition]% for expected > attrition. Calculations used [software and version][; simulation with R replications and seed]. Checklist: - [ ] The primary outcome and the test are named. - [ ] α, sidedness and power are stated. - [ ] The effect size is in the metric the software needs and in raw units. - [ ] The effect size is justified (SESOI, literature or pilot) with a source. - [ ] n is rounded up, inflated for attrition and rounded up again. - [ ] Software and version (and seed and replications for simulation) are given. - [ ] A sensitivity analysis is included if n was constrained. - [ ] Multiplicity is handled if there is more than one primary test. - [ ] No "observed power" calculated after the study. ## Common mistakes and misconceptions | Mistake | Why it is wrong | Fix | |---|---|---| | Using the pilot effect size directly | Pilot estimates are noisy and, after selection, inflated | Use a SESOI, or a safeguard value such as the lower limit of a 60% CI | | Reading `FTestAnovaPower` output as n per group | statsmodels returns total N | Divide by k, or compare with `pwr.anova.test()` | | Reporting N = v from `pwr.f2.test()` | v is the denominator df | N = v + u + 1 (plus any untested covariates) | | Using d instead of d_z for a paired study | Paired power depends on the correlation between measures | Compute d_z from the expected SD of differences | | Forgetting attrition | The analysed n will be below target | Inflate by 1 / (1 - attrition) | | Rounding down | 63.77 per group gives power just under 0.80 | Always round up | | Calculating "observed power" after a non-significant result | It is a function of p and adds no information | Report the CI and a sensitivity analysis instead | | Planning a one-sided test to halve n | Reviewers rarely accept it without a prior reason | Two-sided unless the opposite direction is truly irrelevant and stated in advance | | "Power 0.80 means 80% of studies will be significant" | Only if the assumed effect is the true effect | Say "80% power to detect d = 0.5" | ## Self-test questions 1. Name the four quantities linked in a power analysis. If you know three, what can you compute? 2. Using the normal approximation, roughly how many students per group do you need for d = 0.25 at 80% power? 3. A study planned 64 per group for d = 0.5, but only 40 per group finished. What is the sensitivity analysis question you should now answer? 4. Why is a SESOI usually a better basis for the effect size than a pilot study? 5. statsmodels' `FTestAnovaPower` returned 157.19 for three groups. How many participants per group do you recruit? 6. In a simulation with 2,000 replications, estimated power is 0.80. What is its Monte Carlo standard error? ## My notes and next steps | Item | Your entry | |---|---| | Confidence with analytic power (1 to 5) | [ ] | | Confidence with simulation power (1 to 5) | [ ] | | My study's effect size justification in one sentence | [ ] | | Open questions for my supervisor | [ ] | | Errors logged in stats-error-log | [Link or IDs] | | Next template to study | [stats-analysis-plan] | | Next review date | YYYY-MM-DD | ## Sources and further reading Based on the following sources, cited by name; all text, notation and examples here are original. - Cohen, J. (1988). *Statistical Power Analysis for the Behavioral Sciences* (2nd ed.). Lawrence Erlbaum Associates. - Lakens, D. (2022). Sample size justification. *Collabra: Psychology*, 8(1), 33267. - Hoenig, J. M., and Heisey, D. M. (2001). The abuse of power: the pervasive fallacy of power calculations for data analysis. *The American Statistician*, 55(1), 19–24. - Perugini, M., Gallucci, M., and Costantini, G. (2014). Safeguard power as a protection against imprecise power estimates. *Perspectives on Psychological Science*, 9(3), 319–332. - Albers, C., and Lakens, D. (2018). When power analyses based on pilot data are biased: inaccurate effect size estimators and follow-up bias. *Journal of Experimental Social Psychology*, 74, 187–195. - Champely, S. *pwr: Basic Functions for Power Analysis*, R package version 1.3-0. - Seabold, S., and Perktold, J. (2010). statsmodels: econometric and statistical modeling with Python. *Proceedings of the 9th Python in Science Conference*, 92–96. - Mann, H. B., and Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. *Annals of Mathematical Statistics*, 18(1), 50–60. - NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.2.2 Sample sizes required, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain. - Further reading: de Smith, M. J. (2024). *Statistical Analysis Handbook*, 2024 edition. StatsRef, https://www.statsref.com/; OpenIntro Statistics. ## Answer key **Example B.** (2) Cohen's h. (3) 2 · arcsin(0.7746) - 2 · arcsin(0.6708) = 2 · 0.8861 - 2 · 0.7353 = 0.302. (4) 173 per group. (5) 173 / 0.90 = 192.2, rounded up to 193 per group; 386 total. (6) `power.prop.test()` uses a normal approximation on the raw proportion scale with pooled and unpooled variances, while `pwr.2p.test()` works on the arcsine scale where the variance does not depend on p; the methods differ slightly, and both round to 173. (7) "We will recruit 386 students (193 per group). With 173 per group after 10% attrition, a two-sided test of two proportions at α = .05 has 80% power to detect an increase in the pass rate from 45% to 60% (h = 0.30)." **Exercise C.** (a) `pwr.r.test(r = 0.25, power = 0.90)` gives n = 163.18; the Fisher z formula gives 164.07. Both round up to 164 observations. (b) `pwr.f2.test(u = 4, f2 = 0.10, power = 0.80)` gives v = 119.19, and statsmodels `FTestPowerF2` gives the same denominator df; v rounds up to 120, so N = 120 + 4 + 1 = 125 loaves. The trap is reporting 120 as the sample size. **Self-test.** 1. α, power, effect size and n. Any three determine the fourth: n (sample size calculation), the detectable effect (sensitivity), power (power calculation for a fixed design) or the α needed. 2. n ≈ 15.7 / 0.0625 = 251.2 per group; the exact t-based value is 252.13, so 253 per group. 3. "What effect could 40 per group detect with 80% power?" Here d = 0.634 (`pwr.t.test(n = 40, power = 0.80)`; between 0.6, which needs 45, and 0.8, which needs 26), so an effect of 0.5 is now likely to be missed and a null result is weak evidence of no effect. 4. A SESOI is fixed by what matters, not estimated from a tiny noisy sample; pilot effect sizes are imprecise and, when only promising pilots go ahead, biased upward, which leads to underpowered main studies. 5. 157.19 is the total. 157.19 / 3 = 52.40, so 53 per group, 159 total (plus attrition). 6. √(0.80 · 0.20 / 2000) = √0.00008 = 0.0089, about 0.009. ## Revision history | Version | Date | Author | Change summary | |---|---|---|---| | 0.1 | YYYY-MM-DD | [Your name] | First draft from template | | [0.2] | YYYY-MM-DD | [Your name] | *Example:* Effect size changed from pilot value to SESOI; n recomputed |