# Proportion Tests Method Card: [Your study question or course] > A method card for comparing proportions: the one-proportion test (exact binomial and z), the two-proportion z-test > and its equivalence to the 2x2 chi-square test, Wilson and Newcombe confidence intervals, and the three effect > measures for two groups: risk difference, relative risk and odds ratio. It is for students preparing for an exam, > master's and PhD students with yes/no outcomes in their thesis data, and analysts running A/B tests. It speeds up > learning by putting the decisions in order (which test, which interval, which effect measure), by explaining why R, > SciPy and statsmodels give different p-values for the same counts, and by giving one example fully solved, one half > solved and one left for you. The worked examples use fictional data; replace them with your own. > Source basis: Wilson (1927), Clopper and Pearson (1934), Yates (1934), Newcombe (1998), Agresti and Coull (1998), > the NIST/SEMATECH e-Handbook of Statistical Methods and the R, SciPy and statsmodels documentation, in original words. ## Document control | Field | Value | |---|---| | Course or project | [STAT 101 / thesis chapter 4 / A/B test for a product team] | | Learner or analyst | [Your name] | | Data set | [Brightwell Bakery checkout test, Lakemoor University survey, Harbour City Library returns (all fictional) / your data] | | Software and versions | R 4.5.1 (stats); Python 3.14 with numpy 2.5.4, scipy 1.18.1, statsmodels 0.15.0 | | Date | YYYY-MM-DD | | Version | [0.1] | ## When to use it **One-sentence rule:** use a proportion test when each unit gives a yes/no outcome and you want to compare the share of "yes" with a reference value (one proportion) or between two independent groups (two proportions). | Design | Test | Interval | Effect measure | |---|---|---|---| | One group vs a reference value $p_0$ | Exact binomial test (small n); score z-test (large n) | Wilson (or Clopper-Pearson exact) | $\hat{p} - p_0$ | | Two independent groups | Two-proportion z-test = 2x2 chi-square without correction; Fisher's exact test when expected counts are small | Newcombe for the difference; log-scale CIs for ratios | Risk difference, relative risk, odds ratio | | Same units measured twice (paired yes/no) | McNemar's test | Paired-difference interval | Difference in paired proportions | | Use when | Do not use when | |---|---| | Each unit contributes one independent yes/no outcome | Units contribute several outcomes each (visits per customer) without accounting for it | | Groups are independent (different people) | The same people are in both groups (use McNemar) | | Expected counts of "yes" and "no" are at least about 5 to 10 in each group for z-based methods | Counts are very small (use exact binomial or Fisher's exact test) | | You want a simple comparison | You need to adjust for covariates (use logistic regression) | **Alternatives:** Fisher's exact test and the chi-square test for larger tables (`academic/stats-contingency-tables`); logistic regression for adjustment and several predictors (`academic/stats-logistic-regression`); Bayesian beta-binomial analysis (`academic/stats-bayesian-inference`). ## The question and the data **Research question (A/B test framing):** Brightwell Bakery (fictional) tested a new checkout page for its online orders. Visitors who reached the basket were randomly shown page A (current) or page B (new, with a shorter form). The outcome is whether the visit ended in an order. Did page B change the order rate? | Variable | Role | Measurement level | Values | |---|---|---|---| | `ordered` | Outcome | Binary (nominal) | yes, no | | `page` | Grouping (randomised) | Nominal, 2 levels | A (current), B (new) | | Design | Independent groups, randomised | One visit per visitor, first visit only | n = 300 per page | | Page | Orders (yes) | No order | Total n | Order rate | |---|---|---|---|---| | B (new) | 54 | 246 | 300 | 0.180 | | A (current) | 36 | 264 | 300 | 0.120 | | Total | 90 | 510 | 600 | 0.150 | The chart shows each page's order rate with its 95% Wilson interval. The intervals overlap, yet the difference can still be statistically significant: overlapping individual CIs are not a test of the difference (see Common mistakes). ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "description": "Brightwell Bakery checkout test: order rate by page with 95% Wilson intervals (fictional data)", "width": 300, "height": 220, "data": {"values": [ {"page": "A (current)", "rate": 0.12, "lo": 0.0880, "hi": 0.1617}, {"page": "B (new)", "rate": 0.18, "lo": 0.1407, "hi": 0.2274} ]}, "encoding": {"x": {"field": "page", "type": "nominal", "title": "Checkout page", "axis": {"labelAngle": 0}}}, "layer": [ {"mark": {"type": "bar", "opacity": 0.7, "size": 60}, "encoding": {"y": {"field": "rate", "type": "quantitative", "title": "Order rate", "axis": {"format": "%"}, "scale": {"domain": [0, 0.25]}}}}, {"mark": {"type": "rule", "strokeWidth": 2}, "encoding": {"y": {"field": "lo", "type": "quantitative"}, "y2": {"field": "hi"}}} ] } ``` ## Assumptions and how to check each | Assumption | How to check | R | Python | If violated | |---|---|---|---|---| | Binary outcome, defined in advance | Written definition of "yes" in the plan | Read the plan | Read the plan | Fix the definition before analysis; report any change | | Independent units, one outcome each | Count rows per unit; deduplicate repeat visitors | `any(duplicated(df$visitor_id))` | `df.visitor_id.duplicated().any()` | Keep first exposure only, or use cluster-robust or mixed models | | Independent groups (two-sample tests) | Design: different units in each group | Read the protocol | Read the protocol | Paired data: McNemar's test | | Large enough expected counts for z and chi-square | Expected counts $n \hat{p}_{pooled}$ and $n(1 - \hat{p}_{pooled})$ in each cell | `chisq.test(tab)$expected` | `stats.contingency.expected_freq(tab)` | Exact binomial test or Fisher's exact test | | Fixed sample size (no optional stopping) | Plan states n and when the test is read | Read the plan | Read the plan | Sequential or group-sequential designs; do not stop when p first dips below .05 | | Randomisation worked (A/B tests) | Sample ratio check: observed split vs planned split | `binom.test(n_B, n_A + n_B)` | `stats.binomtest(n_B, n_A + n_B)` | Investigate assignment bugs before reading any result | ## How it works **Plain language.** A sample proportion $\hat{p} = x/n$ estimates the true share $p$. If you repeated the study many times, $\hat{p}$ would vary around $p$ with standard error $\sqrt{p(1-p)/n}$, and for reasonably large n its sampling distribution is close to normal. A z-test divides the observed gap (from $p_0$, or between two groups) by this standard error. For small n you can avoid the normal approximation altogether and compute exact probabilities from the binomial distribution. Intervals need care: the textbook "Wald" interval $\hat{p} \pm 1.96\,SE$ performs poorly for small n or proportions near 0 or 1, which is why the Wilson interval is the recommended default (Agresti and Coull 1998; Brown, Cai and DasGupta 2001). **One proportion.** Exact binomial test: the p-value adds up binomial probabilities of outcomes at least as unlikely as the one observed, under $p = p_0$. Score z-test (uses $p_0$ in the SE): $$ z = \frac{\hat{p} - p_0}{\sqrt{p_0(1 - p_0)/n}} $$ **Wilson interval** (the set of $p_0$ values the score test would not reject), with $z = 1.96$ for 95%: $$ \frac{\hat{p} + \dfrac{z^2}{2n} \pm z\sqrt{\dfrac{\hat{p}(1 - \hat{p})}{n} + \dfrac{z^2}{4n^2}}}{1 + \dfrac{z^2}{n}} $$ **Two proportions.** Under $H_0: p_1 = p_2$ the SE uses the pooled proportion $\bar{p} = (x_1 + x_2)/(n_1 + n_2)$: $$ z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\bar{p}(1 - \bar{p})\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}}, \qquad z^2 = \chi^2_{\text{2x2, no correction}} $$ This identity is the key link: the two-sided two-proportion z-test and the Pearson chi-square test on the 2x2 table without continuity correction are the same test and give the same p-value. **Yates' continuity correction** subtracts 0.5 from each $|O - E|$ before squaring; it makes the test more conservative (larger p-values). R's `prop.test()` and SciPy's `chi2_contingency()` apply it to 2x2 tables by default; statsmodels' `proportions_ztest()` does not. **Newcombe interval** for $\hat{p}_1 - \hat{p}_2$, built from the Wilson limits $(l_1, u_1)$ and $(l_2, u_2)$: $$ \left[\,\hat{d} - \sqrt{(\hat{p}_1 - l_1)^2 + (u_2 - \hat{p}_2)^2},\ \ \hat{d} + \sqrt{(u_1 - \hat{p}_1)^2 + (\hat{p}_2 - l_2)^2}\,\right], \qquad \hat{d} = \hat{p}_1 - \hat{p}_2 $$ **Effect measures** for a 2x2 table with cells $a$ (group 1 yes), $b$ (group 1 no), $c$ (group 2 yes), $d$ (group 2 no). Ratios get their CIs on the log scale, then exponentiate: $$ RD = \hat{p}_1 - \hat{p}_2, \qquad RR = \frac{\hat{p}_1}{\hat{p}_2}, \qquad OR = \frac{a\,d}{b\,c} $$ $$ SE(\ln RR) = \sqrt{\frac{1}{a} - \frac{1}{n_1} + \frac{1}{c} - \frac{1}{n_2}}, \qquad SE(\ln OR) = \sqrt{\frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d}} $$ Cohen's h, $h = 2\arcsin\sqrt{\hat{p}_1} - 2\arcsin\sqrt{\hat{p}_2}$, is the effect size used in power calculations. | Symbol | Meaning | |---|---| | $x$, $n$, $\hat{p}$ | Number of "yes", sample size n, sample proportion | | $p_0$ | Proportion under the null hypothesis | | $\bar{p}$ | Pooled proportion of both groups | | $l$, $u$ | Lower and upper Wilson limits for one group | | $a, b, c, d$ | Cells of the 2x2 table (rows = groups, columns = yes, no) | | $RD$, $RR$, $OR$ | Risk difference, relative risk, odds ratio | | $h$ | Cohen's h | ```mermaid flowchart TD A["Yes/no outcome"] --> B{"How many groups?"} B -->|"One group vs a reference value"| C{"Expected counts at least 10?"} C -->|"No"| D["Exact binomial test + Clopper-Pearson or Wilson CI"] C -->|"Yes"| E["Score z-test + Wilson CI"] B -->|"Two independent groups"| F{"All expected counts at least 5?"} F -->|"No"| G["Fisher's exact test"] F -->|"Yes"| H["Two-proportion z-test = chi-square without correction"] B -->|"Same units twice"| I["McNemar's test"] G --> J["Report RD with Newcombe CI, plus RR or OR with CI"] H --> J B -->|"Need covariates"| K["Logistic regression"] ``` ### R versus Python defaults | Task | R | Python | Same answer? | |---|---|---|---| | Two-proportion test | `prop.test(x, n)`: chi-square **with** Yates correction | `proportions_ztest(count, nobs)`: z **without** correction | No: add `correct = FALSE` in R | | 2x2 chi-square | `chisq.test(tab)`: Yates for 2x2 | `scipy.stats.chi2_contingency(tab)`: Yates for 2x2 | Yes; `correct = FALSE` / `correction=False` to switch off | | One-proportion z | `prop.test(x, n, p = p0)`: score test (SE from $p_0$) with Yates | `proportions_ztest(x, n, value=p0)`: SE from $\hat{p}$ (Wald) | No: use `prop_var=p0` in Python and `correct = FALSE` in R | | Exact binomial | `binom.test(x, n, p = p0)` | `scipy.stats.binomtest(x, n, p=p0)` | Yes | | CI for one proportion | `binom.test`: Clopper-Pearson; `prop.test`: Wilson (with correction unless `correct = FALSE`) | `proportion_confint(x, n)`: **normal (Wald)** by default; set `method="wilson"` | Only if you set the method | | CI for a difference | `prop.test`: Wald interval | `confint_proportions_2indep(..., method="newcomb")` | Compute Newcombe in R by hand (see example A) | ## Worked example A (fully solved): two-proportion z-test for an A/B test **Hypotheses.** $H_0: p_B = p_A$ versus $H_1: p_B \neq p_A$ (two-sided), α = 0.05. Two-sided because the new page could also make things worse, which the bakery would want to know. **Checks.** Visitors were randomised to pages, one visit per visitor, so groups are independent. Expected counts under $H_0$ are 45 orders and 255 non-orders per page, far above 5. The sample ratio is exactly 300 versus 300. The sample size was fixed in advance (no peeking). **R code and output.** ```r x <- c(B = 54, A = 36); n <- c(300, 300) x / n prop.test(x, n) # Yates continuity correction by default pt <- prop.test(x, n, correct = FALSE); pt # equals the two-proportion z-test sqrt(unname(pt$statistic)) # z tab <- matrix(c(54, 246, 36, 264), nrow = 2, byrow = TRUE, dimnames = list(page = c("B", "A"), order = c("yes", "no"))) chisq.test(tab, correct = FALSE)$statistic # Wilson interval per page, then the Newcombe interval for pB - pA wB <- prop.test(54, 300, correct = FALSE)$conf.int; wA <- prop.test(36, 300, correct = FALSE)$conf.int round(rbind(B = wB, A = wA), 4) pB <- 54 / 300; pA <- 36 / 300; rd <- pB - pA lo <- rd - sqrt((pB - wB[1])^2 + (wA[2] - pA)^2) hi <- rd + sqrt((wB[2] - pB)^2 + (pA - wA[1])^2) round(c(rd = rd, lo = lo, hi = hi), 4) # Relative risk and odds ratio with log-scale Wald CIs; Cohen's h rr <- pB / pA; se_lrr <- sqrt(1/54 - 1/300 + 1/36 - 1/300) or <- (54 * 264) / (246 * 36); se_lor <- sqrt(1/54 + 1/246 + 1/36 + 1/264) round(c(rr = rr, lo = exp(log(rr) - 1.96 * se_lrr), hi = exp(log(rr) + 1.96 * se_lrr)), 3) round(c(or = or, lo = exp(log(or) - 1.96 * se_lor), hi = exp(log(or) + 1.96 * se_lor)), 3) 2 * asin(sqrt(pB)) - 2 * asin(sqrt(pA)) ``` ```text B A 0.18 0.12 2-sample test for equality of proportions with continuity correction data: x out of n X-squared = 3.7778, df = 1, p-value = 0.05194 95 percent confidence interval: -0.0002735761 0.1202735761 2-sample test for equality of proportions without continuity correction data: x out of n X-squared = 4.2353, df = 1, p-value = 0.03959 95 percent confidence interval: 0.003059757 0.116940243 [1] 2.057983 X-squared 4.235294 [,1] [,2] B 0.1407 0.2274 A 0.0880 0.1617 rd lo hi 0.0600 0.0027 0.1172 rr lo hi 1.500 1.015 2.216 or lo hi 1.61 1.02 2.54 [1] 0.1688148 ``` **Python code and output.** ```python import numpy as np from scipy import stats from statsmodels.stats.proportion import (proportions_ztest, proportion_confint, confint_proportions_2indep) count = np.array([54, 36]); nobs = np.array([300, 300]) # B, A z, p = proportions_ztest(count, nobs) # pooled SE, no continuity correction print(round(z, 4), round(p, 5)) tab = np.array([[54, 246], [36, 264]]) chi2, p_chi, dof, _ = stats.chi2_contingency(tab) # SciPy applies Yates to 2x2 by default print(round(chi2, 4), round(p_chi, 5)) chi2, p_chi, dof, _ = stats.chi2_contingency(tab, correction=False) print(round(chi2, 4), round(p_chi, 5)) print(proportion_confint(count, nobs, method="wilson")) print(confint_proportions_2indep(54, 300, 36, 300, method="newcomb", compare="diff")) print(stats.contingency.relative_risk(54, 300, 36, 300).confidence_interval()) res = stats.contingency.odds_ratio(tab, kind="sample") print(res.statistic, res.confidence_interval()) ``` ```text 2.058 0.03959 3.7778 0.05194 4.2353 0.03959 (array([0.1406583 , 0.08795084]), array([0.2274332 , 0.16165781])) (np.float64(0.0027012862312462207), np.float64(0.11724558268991167)) ConfidenceInterval(low=np.float64(1.0154088234388892), high=np.float64(2.2158562620914752)) 1.6097560975609757 ConfidenceInterval(low=np.float64(1.0202006961551373), high=np.float64(2.540004827874271)) ``` **Calculation by hand.** 1. Proportions: $\hat{p}_B = 54/300 = 0.18$, $\hat{p}_A = 36/300 = 0.12$, $\hat{d} = 0.06$. 2. Pooled: $\bar{p} = 90/600 = 0.15$; $SE_0 = \sqrt{0.15 \times 0.85 \times (1/300 + 1/300)} = \sqrt{0.00085} = 0.02915$. 3. $z = 0.06/0.02915 = 2.058$, two-sided p = .040, and $z^2 = 4.235$ = the uncorrected chi-square. Same test. 4. Wilson for page B ($\hat{p} = 0.18$, $n = 300$, $z^2 = 3.8416$): centre $(0.18 + 3.8416/600)/(1 + 3.8416/300) = 0.18640/1.01281 = 0.1840$; half-width $1.96\sqrt{0.18 \times 0.82/300 + 3.8416/360000}/1.01281 = 0.0434$; interval [0.1407, 0.2274]. Notice that the Wilson interval is centred slightly closer to 0.5 than $\hat{p}$ is. 5. Newcombe: lower $0.06 - \sqrt{(0.18 - 0.1407)^2 + (0.1617 - 0.12)^2} = 0.06 - 0.0573 = 0.0027$; upper $0.06 + \sqrt{(0.2274 - 0.18)^2 + (0.12 - 0.0880)^2} = 0.06 + 0.0572 = 0.1172$. 6. $RR = 0.18/0.12 = 1.50$; $OR = (54 \times 264)/(246 \times 36) = 14256/8856 = 1.61$. **The continuity correction lesson.** With Yates' correction (R's `prop.test` default and SciPy's `chi2_contingency` default) the p-value is .052; without it (statsmodels' default, and the plain z-test) it is .040. Neither is "the right one" after the fact. Yates' correction makes the test conservative: an exact calculation for 300 visitors per page and a true order rate of 15% in both groups gives a Type I error rate of about 5.0% for the uncorrected test and about 3.8% for the corrected one, so here the uncorrected test is the one that stays close to the nominal 5%, which is why many statisticians prefer it for samples of this size. Decide in the analysis plan, report which you used, and never choose the version that gives the p-value you prefer. **Effect sizes.** Risk difference 6.0 percentage points, Newcombe 95% CI [0.3, 11.7]; relative risk 1.50, 95% CI [1.02, 2.22] (orders were 50% more likely with page B); odds ratio 1.61, 95% CI [1.02, 2.54]; Cohen's h = 0.17. For decisions, the risk difference is the most useful: out of 1,000 visitors who reach the basket, page B would bring about 60 more orders, with a plausible range of 3 to 117. **Interpretation and report.** Page B had a higher order rate than page A (18.0% versus 12.0%, *n* = 300 each), a difference of 6.0 percentage points, Newcombe 95% CI [0.3, 11.7], *z* = 2.06, *p* = .040 (two-proportion z-test without continuity correction; *χ*²(1) = 4.24), relative risk = 1.50, 95% CI [1.02, 2.22]. **A/B test caveat: power.** A plan for this test should have fixed the sample size from the smallest difference worth detecting. To detect 12% versus 18% with 80% power at α = 0.05 you need about 555 visitors per page (`power.prop.test(p1 = 0.12, p2 = 0.18, power = 0.80)` in R gives n = 554.8; statsmodels' `NormalIndPower` with Cohen's h = 0.169 gives 550.8, which differs slightly because it uses the arcsine approximation). With 300 per page this test was underpowered, so the true effect could easily be much smaller than 6 points (the CI reaches 0.3). Use `academic/stats-power-sample-size` and `product/ab-test-plan` before the next test. ## Worked example B (half solved): one-proportion test **Setting.** Lakemoor University's library claims that half of all students use it at least weekly. A survey of 40 randomly sampled students finds that 27 do. Is the true share different from 0.5? ```r binom.test(27, 40, p = 0.5) # exact binomial, Clopper-Pearson CI prop.test(27, 40, p = 0.5, correct = FALSE) # score test, Wilson CI ``` ```python from scipy import stats from statsmodels.stats.proportion import proportions_ztest, proportion_confint r = stats.binomtest(27, 40, p=0.5); print(r.pvalue, r.proportion_ci(method="exact")) print(proportions_ztest(27, 40, value=0.5)) # SE from the sample proportion (Wald) print(proportions_ztest(27, 40, value=0.5, prop_var=0.5)) # SE from p0 (score test) print(proportion_confint(27, 40, method="wilson")) print(proportion_confint(27, 40, method="normal")) ``` ```text # Trimmed output, R then Python Exact binomial test: number of successes = 27, number of trials = 40, p-value = 0.03848 95 percent confidence interval: 0.5087051 0.8142710 1-sample proportions test without continuity correction: X-squared = 4.9, df = 1, p-value = 0.02686 95 percent confidence interval: 0.5201775 0.7991550 0.03847730828420026 ConfidenceInterval(low=np.float64(0.5087051245215517), high=np.float64(0.8142710332366502)) (np.float64(2.363058523195223), np.float64(0.018124807150689443)) (np.float64(2.213594362117866), np.float64(0.026856695507524384)) (0.5201774618987767, 0.7991549776821731) (0.5298517973940279, 0.8201482026059722) ``` Complete the steps: 1. $\hat{p} =$ [ ]. Expected counts under $H_0$: $40 \times 0.5 =$ [ ] yes and [ ] no, so the z approximation is [acceptable / doubtful]. 2. Score z: $z = (0.675 - 0.5)/\sqrt{0.5 \times 0.5/40} = 0.175/$[ ] $=$ [ ], so $z^2 =$ [ ], matching R's X-squared. 3. Three p-values for the same data: exact [ ], score z [ ], statsmodels default (Wald SE) [ ]. The Wald version is smallest because it uses $\hat{p} = 0.675$ in the SE, and $0.675 \times 0.325$ is [smaller / larger] than $0.5 \times 0.5$. 4. Three 95% CIs: Clopper-Pearson [ ], Wilson [ ], Wald [ ]. The recommended default for reporting is [ ]. 5. Report: [ ] of 40 students ([ ]%) used the library weekly, 95% Wilson CI [ ], which [is / is not] consistent with the claimed 50% (exact binomial test, *p* = [ ]). ## Exercise C (unsolved): two branches, overdue returns Harbour City Library (fictional) compared the share of loans returned late at two branches in one month: Central had 18 late returns out of 120 loans; North had 9 late out of 110. | Branch | Late | On time | Total | |---|---|---|---| | Central | 18 | 102 | 120 | | North | 9 | 101 | 110 | Tasks: 1. State $H_0$ and $H_1$ and check the expected counts. 2. Compute $\hat{p}_1$, $\hat{p}_2$, the pooled proportion, the SE under $H_0$, z and the two-sided p-value by hand. 3. Run `prop.test(c(18, 9), c(120, 110), correct = FALSE)` in R and `proportions_ztest([18, 9], [120, 110])` in Python. Then run R's default `prop.test(c(18, 9), c(120, 110))` and explain why its p-value is larger. 4. Compute the risk difference with a Newcombe 95% CI, and the relative risk and odds ratio with 95% CIs. 5. Write the APA-style result. Does the result show that the branches have the same late-return rate? ## Reading software output | Output line | Meaning | Report it? | |---|---|---| | `2-sample test for equality of proportions with continuity correction` (R) | Yates-corrected chi-square; the default in `prop.test` | Name the test and the correction | | `X-squared = 4.2353, df = 1` | Chi-square statistic; equals $z^2$ when uncorrected | Yes: *χ*²(1) = 4.24, or *z* = 2.06 | | `p-value = 0.03959` | Two-sided p-value | Yes: *p* = .040 | | `95 percent confidence interval` in two-sample `prop.test` | Wald interval for $p_1 - p_2$ (with correction if applied) | Prefer Newcombe; say which you report | | `95 percent confidence interval` in `binom.test` | Clopper-Pearson exact interval for one proportion | Yes, or report Wilson | | `(z, p)` tuple from `proportions_ztest` | z and two-sided p; no continuity correction; one-sample SE uses $\hat{p}$ unless `prop_var` is set | Yes, after checking the variant | | `proportion_confint(...)` | Interval for one proportion; default `method="normal"` (Wald) | Set `method="wilson"` | | `relative_risk(...).confidence_interval()`, `odds_ratio(...)` | Ratio effect sizes with CIs (log-scale for RR; conditional MLE by default for OR in SciPy, sample OR when `kind="sample"`) | Yes, name the method | ## Reporting template **One proportion:** [x] of [n] [units] ([ ]%) [had the outcome], 95% Wilson CI [[ ], [ ]]; this [differs / does not differ] from the reference value of [p0] ([exact binomial / score z] test, *p* = [ ]). **Two proportions:** [Outcome] occurred in [x1] of [n1] ([ ]%) in [group 1] and [x2] of [n2] ([ ]%) in [group 2], a difference of [ ] percentage points, Newcombe 95% CI [[ ], [ ]], *z* = [ ], *p* = [ ] ([with / without] continuity correction), relative risk = [ ], 95% CI [[ ], [ ]]. | Group | Yes | n | Proportion [95% Wilson CI] | Difference [95% Newcombe CI] | Test | p | RR [95% CI] | OR [95% CI] | |---|---|---|---|---|---|---|---|---| | *Example:* Page B | 54 | 300 | 0.180 [0.141, 0.227] | 0.060 [0.003, 0.117] | z = 2.06, no correction | .040 | 1.50 [1.02, 2.22] | 1.61 [1.02, 2.54] | | *Example:* Page A | 36 | 300 | 0.120 [0.088, 0.162] | reference | | | | | | [Group] | [ ] | [ ] | [ ] | [ ] | [ ] | [ ] | [ ] | [ ] | **Results checklist:** - [ ] The "yes" outcome and the unit of analysis are defined (one outcome per unit). - [ ] Counts and denominators are reported for every group, not only percentages. - [ ] The test variant is named: exact or z; with or without continuity correction; score or Wald SE. - [ ] Expected counts were checked; exact or Fisher's test used when they are small. - [ ] The CI method is named (Wilson, Clopper-Pearson, Newcombe) and the Wald interval is avoided for small n. - [ ] At least one effect measure with a CI is reported (risk difference for decisions; RR or OR as appropriate). - [ ] For A/B tests: planned sample size, randomisation check (sample ratio) and no optional stopping are reported. - [ ] Odds ratios are not described as "times more likely" unless the outcome is rare. ## Common mistakes and misconceptions - **Not knowing the continuity correction default.** R's `prop.test` and SciPy's `chi2_contingency` apply Yates to 2x2 tables; statsmodels' `proportions_ztest` does not. In Worked example A this moves p from .040 to .052. - **Choosing the variant after seeing the p-value.** Picking whichever of .040 and .052 suits your story is p-hacking. Fix the method in the plan. - **Using the Wald interval for small n.** $\hat{p} \pm 1.96\,SE$ can extend below 0 or above 1 and covers the true value less often than 95% of the time. Use Wilson. Note that statsmodels' `proportion_confint` defaults to the Wald (`"normal"`) method. - **"The two CIs overlap, so there is no significant difference."** Wrong. In Worked example A the Wilson intervals overlap (0.141 to 0.227 and 0.088 to 0.162), yet the difference is significant (p = .040). Compare using the CI of the difference. - **Reading an odds ratio as a relative risk.** OR = 1.61 does not mean "61% more likely" here; the relative risk is 1.50. The two are close only when the outcome is rare (under about 10%). - **Peeking and stopping early in A/B tests.** Checking the p-value every day and stopping when it first drops below .05 inflates the false positive rate far above 5%. Fix n in advance or use a sequential design. - **Counting visits instead of visitors.** If a visitor returns five times, their five outcomes are not independent. Analyse one outcome per randomised unit. - **Using a two-sample test for paired yes/no data.** Before-and-after yes/no answers from the same people need McNemar's test. - **"p = 0.04 means a 4% chance that page B is no better."** No: it is the probability of a difference at least this large if the pages truly had equal order rates. ## Self-test questions 1. You run `prop.test(c(54, 36), c(300, 300))` in R and `proportions_ztest([54, 36], [300, 300])` in Python and get p = .052 and p = .040. Explain the difference and how to make them agree. 2. Show the relationship between the two-proportion z statistic and the 2x2 chi-square statistic. When does it hold? 3. Why is the Wilson interval preferred over the Wald interval for a proportion such as 3 out of 25? 4. In a study, 30% of the exposed and 20% of the unexposed have the outcome. Compute the risk difference, the relative risk and the odds ratio. Which one would you give a manager, and why? 5. Interleaving check: name the test. (a) 14 of 20 Brightwell Bakery loaves pass a quality check; target pass rate 90%. (b) 120 Lakemoor students answer "yes/no" to a question before and after a lecture. (c) Click rates on two randomised newsletter versions, 2,000 recipients each. (d) Exam pass rates across four tutorial groups. 6. Before an A/B test, why should you calculate the sample size, and which quantity drives it most? ## My notes and next steps | Item | My entry | |---|---| | Confidence with proportion tests (1 = lost, 5 = could teach it) | [ ] | | Test variant and CI method I will use for my data, and why | [ ] | | Software arguments I must set explicitly (`correct`, `method`, `prop_var`) | [ ] | | Open questions for my tutor, supervisor or product team | [ ] | | Mistakes logged in `academic/stats-error-log` | [Date and short description] | | Next review date (spacing: +2 days, +1 week, +1 month) | YYYY-MM-DD | ## Sources and further reading Structure aligned with the NIST/SEMATECH e-Handbook of Statistical Methods and the reporting elements named in APA JARS-Quant; all wording, data and examples here are original. - Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. *Journal of the American Statistical Association*, 22(158), 209–212. - Clopper, C. J., and Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. *Biometrika*, 26(4), 404–413. - Yates, F. (1934). Contingency tables involving small numbers and the χ² test. *Supplement to the Journal of the Royal Statistical Society*, 1(2), 217–235. - Newcombe, R. G. (1998). Interval estimation for the difference between independent proportions: comparison of eleven methods. *Statistics in Medicine*, 17(8), 873–890. - Agresti, A., and Coull, B. A. (1998). Approximate is better than "exact" for interval estimation of binomial proportions. *The American Statistician*, 52(2), 119–126. - Brown, L. D., Cai, T. T., and DasGupta, A. (2001). Interval estimation for a binomial proportion. *Statistical Science*, 16(2), 101–133. - Cohen, J. (1988). *Statistical Power Analysis for the Behavioral Sciences* (2nd ed.). Lawrence Erlbaum. - Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. *The American Statistician*, 70(2), 129–133. - NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.4 Does the proportion of defectives meet requirements?, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain. - R Core Team. `prop.test`, `binom.test`, `chisq.test`, `power.prop.test` documentation, R 4.5.1. statsmodels documentation for `proportions_ztest`, `proportion_confint` and `confint_proportions_2indep` (0.15); SciPy documentation for `binomtest`, `chi2_contingency`, `relative_risk` and `odds_ratio` (1.18). - Further reading (by name only): de Smith, M. J., *Statistical Analysis Handbook* (2024 edition, statsref.com); OpenIntro Statistics; Agresti, A., *Categorical Data Analysis*; Kohavi, Tang and Xu, *Trustworthy Online Controlled Experiments*; American Psychological Association, *Publication Manual* (7th ed.) and JARS-Quant. ## Answer key **Worked example B.** (1) $\hat{p} = 27/40 = 0.675$; expected 20 yes and 20 no, so the z approximation is acceptable. (2) $\sqrt{0.25/40} = 0.0791$, $z = 0.175/0.0791 = 2.21$, $z^2 = 4.90$. (3) Exact .038, score z .027, Wald-SE z .018 (z = 2.36); $0.675 \times 0.325 = 0.219$ is smaller than 0.25, so the Wald SE is smaller and z larger. (4) Clopper-Pearson [0.509, 0.814], Wilson [0.520, 0.799], Wald [0.530, 0.820]; report Wilson (or Clopper-Pearson when an exact, conservative interval is required). (5) 27 of 40 students (67.5%) used the library weekly, 95% Wilson CI [52.0%, 79.9%], which is not consistent with the claimed 50% (exact binomial test, *p* = .038). **Exercise C.** (1) $H_0: p_{Central} = p_{North}$, $H_1: p_{Central} \neq p_{North}$; pooled $\bar{p} = 27/230 = 0.117$, expected late returns 14.1 and 12.9, all expected counts well above 5. (2) $\hat{p}_1 = 0.150$, $\hat{p}_2 = 0.082$; $SE_0 = \sqrt{0.117 \times 0.883 \times (1/120 + 1/110)} = 0.0425$; $z = 0.0682/0.0425 = 1.60$, p = .109. (3) R uncorrected: X-squared = 2.575, p = .1086; Python: z = 1.605, p = .1086. R's default applies Yates' correction, which shrinks the chi-square and so gives a larger p. (4) RD = 0.068, Newcombe 95% CI [-0.017, 0.152]; RR = 1.83, 95% CI [0.86, 3.91]; OR = 1.98, 95% CI [0.85, 4.62]. (5) "Late returns occurred in 18 of 120 loans (15.0%) at Central and 9 of 110 (8.2%) at North, a difference of 6.8 percentage points, Newcombe 95% CI [-1.7, 15.2], *z* = 1.60, *p* = .109 (no continuity correction), relative risk = 1.83, 95% CI [0.86, 3.91]." No: the data are compatible with no difference but also with Central having almost four times North's rate. The study is too small to tell; a planned larger sample (or pooling several months) is needed. **Self-test.** 1. R's `prop.test` applies Yates' continuity correction by default; `proportions_ztest` does not. Use `prop.test(..., correct = FALSE)` in R; both then give p = .040. 2. $z^2 = \chi^2$ for the 2x2 table when the z-test uses the pooled SE, the chi-square is Pearson's without continuity correction, and the z-test is two-sided. 3. With $\hat{p} = 0.12$ and n = 25 the Wald interval is $0.12 \pm 0.127$, which runs below 0 (statsmodels silently clips it at 0), and its actual coverage is well under 95%. The Wilson interval stays inside [0, 1], is asymmetric where it should be, and has coverage close to 95%. 4. RD = 0.10 (10 percentage points); RR = 0.30/0.20 = 1.5; OR = (0.30/0.70)/(0.20/0.80) = 0.4286/0.25 = 1.71. For a manager, the risk difference: "10 more cases per 100 people" is concrete; ratios hide the baseline. 5. (a) One-proportion test against 0.90; expected failures are 20 x 0.10 = 2, so use the exact binomial test. (b) McNemar's test (paired yes/no). (c) Two-proportion z-test (or chi-square without correction); with n = 2,000 per arm the correction hardly matters. (d) Chi-square test of independence on a 4x2 table (`academic/stats-contingency-tables`). 6. To have a good chance (power) of detecting the smallest effect worth acting on, and to fix the stopping point in advance so that peeking cannot inflate false positives. The minimum detectable difference drives it most: halving the difference roughly quadruples the required sample size. ## Revision history | Version | Date | Author | Change summary | |---|---|---|---| | 0.1 | YYYY-MM-DD | [Your name] | First draft from the template | | 0.2 | YYYY-MM-DD | [Your name] | *Example:* Replaced checkout example with my A/B test; correction choice fixed in the plan |