Proportion Tests Method Card Template (Statistics, R and Python)

Proportion Tests Method Card: [Your study question or course]

A method card for comparing proportions: the one-proportion test (exact binomial and z), the two-proportion z-test and its equivalence to the 2x2 chi-square test, Wilson and Newcombe confidence intervals, and the three effect measures for two groups: risk difference, relative risk and odds ratio. It is for students preparing for an exam, master's and PhD students with yes/no outcomes in their thesis data, and analysts running A/B tests. It speeds up learning by putting the decisions in order (which test, which interval, which effect measure), by explaining why R, SciPy and statsmodels give different p-values for the same counts, and by giving one example fully solved, one half solved and one left for you. The worked examples use fictional data; replace them with your own. Source basis: Wilson (1927), Clopper and Pearson (1934), Yates (1934), Newcombe (1998), Agresti and Coull (1998), the NIST/SEMATECH e-Handbook of Statistical Methods and the R, SciPy and statsmodels documentation, in original words.

Document control

When to use it

One-sentence rule: use a proportion test when each unit gives a yes/no outcome and you want to compare the share of "yes" with a reference value (one proportion) or between two independent groups (two proportions).

Alternatives: Fisher's exact test and the chi-square test for larger tables (academic/stats-contingency-tables); logistic regression for adjustment and several predictors (academic/stats-logistic-regression); Bayesian beta-binomial analysis (academic/stats-bayesian-inference).

The question and the data

Research question (A/B test framing): Brightwell Bakery (fictional) tested a new checkout page for its online orders. Visitors who reached the basket were randomly shown page A (current) or page B (new, with a shorter form). The outcome is whether the visit ended in an order. Did page B change the order rate?

The chart shows each page's order rate with its 95% Wilson interval. The intervals overlap, yet the difference can still be statistically significant: overlapping individual CIs are not a test of the difference (see Common mistakes).

Assumptions and how to check each

How it works

Plain language. A sample proportion $\hat{p} = x/n$ estimates the true share $p$. If you repeated the study many times, $\hat{p}$ would vary around $p$ with standard error $\sqrt{p(1-p)/n}$, and for reasonably large n its sampling distribution is close to normal. A z-test divides the observed gap (from $p_0$, or between two groups) by this standard error. For small n you can avoid the normal approximation altogether and compute exact probabilities from the binomial distribution. Intervals need care: the textbook "Wald" interval $\hat{p} \pm 1.96\,SE$ performs poorly for small n or proportions near 0 or 1, which is why the Wilson interval is the recommended default (Agresti and Coull 1998; Brown, Cai and DasGupta 2001).

One proportion. Exact binomial test: the p-value adds up binomial probabilities of outcomes at least as unlikely as the one observed, under $p = p_0$. Score z-test (uses $p_0$ in the SE):

Wilson interval (the set of $p_0$ values the score test would not reject), with $z = 1.96$ for 95%:

Two proportions. Under $H_0: p_1 = p_2$ the SE uses the pooled proportion $\bar{p} = (x_1 + x_2)/(n_1 + n_2)$:

This identity is the key link: the two-sided two-proportion z-test and the Pearson chi-square test on the 2x2 table without continuity correction are the same test and give the same p-value. Yates' continuity correction subtracts 0.5 from each $|O - E|$ before squaring; it makes the test more conservative (larger p-values). R's prop.test() and SciPy's chi2_contingency() apply it to 2x2 tables by default; statsmodels' proportions_ztest() does not.

Newcombe interval for $\hat{p}_1 - \hat{p}_2$, built from the Wilson limits $(l_1, u_1)$ and $(l_2, u_2)$:

Effect measures for a 2x2 table with cells $a$ (group 1 yes), $b$ (group 1 no), $c$ (group 2 yes), $d$ (group 2 no). Ratios get their CIs on the log scale, then exponentiate:

Cohen's h, $h = 2\arcsin\sqrt{\hat{p}_1} - 2\arcsin\sqrt{\hat{p}_2}$, is the effect size used in power calculations.

R versus Python defaults

Worked example A (fully solved): two-proportion z-test for an A/B test

Hypotheses. $H_0: p_B = p_A$ versus $H_1: p_B \neq p_A$ (two-sided), α = 0.05. Two-sided because the new page could also make things worse, which the bakery would want to know.

Checks. Visitors were randomised to pages, one visit per visitor, so groups are independent. Expected counts under $H_0$ are 45 orders and 255 non-orders per page, far above 5. The sample ratio is exactly 300 versus 300. The sample size was fixed in advance (no peeking).

R code and output.

x <- c(B = 54, A = 36); n <- c(300, 300)
x / n
prop.test(x, n)                              # Yates continuity correction by default
pt <- prop.test(x, n, correct = FALSE); pt   # equals the two-proportion z-test
sqrt(unname(pt$statistic))                   # z
tab <- matrix(c(54, 246, 36, 264), nrow = 2, byrow = TRUE,
              dimnames = list(page = c("B", "A"), order = c("yes", "no")))
chisq.test(tab, correct = FALSE)$statistic
# Wilson interval per page, then the Newcombe interval for pB - pA
wB <- prop.test(54, 300, correct = FALSE)$conf.int; wA <- prop.test(36, 300, correct = FALSE)$conf.int
round(rbind(B = wB, A = wA), 4)
pB <- 54 / 300; pA <- 36 / 300; rd <- pB - pA
lo <- rd - sqrt((pB - wB[1])^2 + (wA[2] - pA)^2)
hi <- rd + sqrt((wB[2] - pB)^2 + (pA - wA[1])^2)
round(c(rd = rd, lo = lo, hi = hi), 4)
# Relative risk and odds ratio with log-scale Wald CIs; Cohen's h
rr <- pB / pA; se_lrr <- sqrt(1/54 - 1/300 + 1/36 - 1/300)
or <- (54 * 264) / (246 * 36); se_lor <- sqrt(1/54 + 1/246 + 1/36 + 1/264)
round(c(rr = rr, lo = exp(log(rr) - 1.96 * se_lrr), hi = exp(log(rr) + 1.96 * se_lrr)), 3)
round(c(or = or, lo = exp(log(or) - 1.96 * se_lor), hi = exp(log(or) + 1.96 * se_lor)), 3)
2 * asin(sqrt(pB)) - 2 * asin(sqrt(pA))

   B    A
0.18 0.12

	2-sample test for equality of proportions with continuity correction

data:  x out of n
X-squared = 3.7778, df = 1, p-value = 0.05194
95 percent confidence interval:
 -0.0002735761  0.1202735761

	2-sample test for equality of proportions without continuity correction

data:  x out of n
X-squared = 4.2353, df = 1, p-value = 0.03959
95 percent confidence interval:
 0.003059757 0.116940243

[1] 2.057983
X-squared
 4.235294
    [,1]   [,2]
B 0.1407 0.2274
A 0.0880 0.1617
    rd     lo     hi
0.0600 0.0027 0.1172
   rr    lo    hi
1.500 1.015 2.216
  or   lo   hi
1.61 1.02 2.54
[1] 0.1688148

Python code and output.

import numpy as np
from scipy import stats
from statsmodels.stats.proportion import (proportions_ztest, proportion_confint,
                                          confint_proportions_2indep)

count = np.array([54, 36]); nobs = np.array([300, 300])    # B, A
z, p = proportions_ztest(count, nobs)                      # pooled SE, no continuity correction
print(round(z, 4), round(p, 5))
tab = np.array([[54, 246], [36, 264]])
chi2, p_chi, dof, _ = stats.chi2_contingency(tab)          # SciPy applies Yates to 2x2 by default
print(round(chi2, 4), round(p_chi, 5))
chi2, p_chi, dof, _ = stats.chi2_contingency(tab, correction=False)
print(round(chi2, 4), round(p_chi, 5))
print(proportion_confint(count, nobs, method="wilson"))
print(confint_proportions_2indep(54, 300, 36, 300, method="newcomb", compare="diff"))
print(stats.contingency.relative_risk(54, 300, 36, 300).confidence_interval())
res = stats.contingency.odds_ratio(tab, kind="sample")
print(res.statistic, res.confidence_interval())

2.058 0.03959
3.7778 0.05194
4.2353 0.03959
(array([0.1406583 , 0.08795084]), array([0.2274332 , 0.16165781]))
(np.float64(0.0027012862312462207), np.float64(0.11724558268991167))
ConfidenceInterval(low=np.float64(1.0154088234388892), high=np.float64(2.2158562620914752))
1.6097560975609757 ConfidenceInterval(low=np.float64(1.0202006961551373), high=np.float64(2.540004827874271))

Calculation by hand.

1. Proportions: $\hat{p}_B = 54/300 = 0.18$, $\hat{p}_A = 36/300 = 0.12$, $\hat{d} = 0.06$.
2. Pooled: $\bar{p} = 90/600 = 0.15$; $SE_0 = \sqrt{0.15 \times 0.85 \times (1/300 + 1/300)} = \sqrt{0.00085} = 0.02915$.
3. $z = 0.06/0.02915 = 2.058$, two-sided p = .040, and $z^2 = 4.235$ = the uncorrected chi-square. Same test.
4. Wilson for page B ($\hat{p} = 0.18$, $n = 300$, $z^2 = 3.8416$): centre $(0.18 + 3.8416/600)/(1 + 3.8416/300) = 0.18640/1.01281 = 0.1840$; half-width $1.96\sqrt{0.18 \times 0.82/300 + 3.8416/360000}/1.01281 = 0.0434$; interval [0.1407, 0.2274]. Notice that the Wilson interval is centred slightly closer to 0.5 than $\hat{p}$ is.
5. Newcombe: lower $0.06 - \sqrt{(0.18 - 0.1407)^2 + (0.1617 - 0.12)^2} = 0.06 - 0.0573 = 0.0027$; upper $0.06 + \sqrt{(0.2274 - 0.18)^2 + (0.12 - 0.0880)^2} = 0.06 + 0.0572 = 0.1172$.
6. $RR = 0.18/0.12 = 1.50$; $OR = (54 \times 264)/(246 \times 36) = 14256/8856 = 1.61$.

The continuity correction lesson. With Yates' correction (R's prop.test default and SciPy's chi2_contingency default) the p-value is .052; without it (statsmodels' default, and the plain z-test) it is .040. Neither is "the right one" after the fact. Yates' correction makes the test conservative: an exact calculation for 300 visitors per page and a true order rate of 15% in both groups gives a Type I error rate of about 5.0% for the uncorrected test and about 3.8% for the corrected one, so here the uncorrected test is the one that stays close to the nominal 5%, which is why many statisticians prefer it for samples of this size. Decide in the analysis plan, report which you used, and never choose the version that gives the p-value you prefer.

Effect sizes. Risk difference 6.0 percentage points, Newcombe 95% CI [0.3, 11.7]; relative risk 1.50, 95% CI [1.02, 2.22] (orders were 50% more likely with page B); odds ratio 1.61, 95% CI [1.02, 2.54]; Cohen's h = 0.17. For decisions, the risk difference is the most useful: out of 1,000 visitors who reach the basket, page B would bring about 60 more orders, with a plausible range of 3 to 117.

Interpretation and report. Page B had a higher order rate than page A (18.0% versus 12.0%, n = 300 each), a difference of 6.0 percentage points, Newcombe 95% CI [0.3, 11.7], z = 2.06, p = .040 (two-proportion z-test without continuity correction; χ²(1) = 4.24), relative risk = 1.50, 95% CI [1.02, 2.22].

A/B test caveat: power. A plan for this test should have fixed the sample size from the smallest difference worth detecting. To detect 12% versus 18% with 80% power at α = 0.05 you need about 555 visitors per page (power.prop.test(p1 = 0.12, p2 = 0.18, power = 0.80) in R gives n = 554.8; statsmodels' NormalIndPower with Cohen's h = 0.169 gives 550.8, which differs slightly because it uses the arcsine approximation). With 300 per page this test was underpowered, so the true effect could easily be much smaller than 6 points (the CI reaches 0.3). Use academic/stats-power-sample-size and product/ab-test-plan before the next test.

Worked example B (half solved): one-proportion test

Setting. Lakemoor University's library claims that half of all students use it at least weekly. A survey of 40 randomly sampled students finds that 27 do. Is the true share different from 0.5?

binom.test(27, 40, p = 0.5)                     # exact binomial, Clopper-Pearson CI
prop.test(27, 40, p = 0.5, correct = FALSE)     # score test, Wilson CI

from scipy import stats
from statsmodels.stats.proportion import proportions_ztest, proportion_confint
r = stats.binomtest(27, 40, p=0.5); print(r.pvalue, r.proportion_ci(method="exact"))
print(proportions_ztest(27, 40, value=0.5))                  # SE from the sample proportion (Wald)
print(proportions_ztest(27, 40, value=0.5, prop_var=0.5))    # SE from p0 (score test)
print(proportion_confint(27, 40, method="wilson"))
print(proportion_confint(27, 40, method="normal"))

# Trimmed output, R then Python
Exact binomial test: number of successes = 27, number of trials = 40, p-value = 0.03848
  95 percent confidence interval: 0.5087051 0.8142710
1-sample proportions test without continuity correction: X-squared = 4.9, df = 1, p-value = 0.02686
  95 percent confidence interval: 0.5201775 0.7991550

0.03847730828420026 ConfidenceInterval(low=np.float64(0.5087051245215517), high=np.float64(0.8142710332366502))
(np.float64(2.363058523195223), np.float64(0.018124807150689443))
(np.float64(2.213594362117866), np.float64(0.026856695507524384))
(0.5201774618987767, 0.7991549776821731)
(0.5298517973940279, 0.8201482026059722)

Complete the steps:

1. $\hat{p} =$ [ ]. Expected counts under $H_0$: $40 \times 0.5 =$ [ ] yes and [ ] no, so the z approximation is [acceptable / doubtful].
2. Score z: $z = (0.675 - 0.5)/\sqrt{0.5 \times 0.5/40} = 0.175/$[ ] $=$ [ ], so $z^2 =$ [ ], matching R's X-squared.
3. Three p-values for the same data: exact [ ], score z [ ], statsmodels default (Wald SE) [ ]. The Wald version is smallest because it uses $\hat{p} = 0.675$ in the SE, and $0.675 \times 0.325$ is [smaller / larger] than $0.5 \times 0.5$.
4. Three 95% CIs: Clopper-Pearson [ ], Wilson [ ], Wald [ ]. The recommended default for reporting is [ ].
5. Report: [ ] of 40 students ([ ]%) used the library weekly, 95% Wilson CI [ ], which [is / is not] consistent with the claimed 50% (exact binomial test, p = [ ]).

Exercise C (unsolved): two branches, overdue returns

Harbour City Library (fictional) compared the share of loans returned late at two branches in one month: Central had 18 late returns out of 120 loans; North had 9 late out of 110.

Tasks:

1. State $H_0$ and $H_1$ and check the expected counts.
2. Compute $\hat{p}_1$, $\hat{p}_2$, the pooled proportion, the SE under $H_0$, z and the two-sided p-value by hand.
3. Run prop.test(c(18, 9), c(120, 110), correct = FALSE) in R and proportions_ztest([18, 9], [120, 110]) in Python. Then run R's default prop.test(c(18, 9), c(120, 110)) and explain why its p-value is larger.
4. Compute the risk difference with a Newcombe 95% CI, and the relative risk and odds ratio with 95% CIs.
5. Write the APA-style result. Does the result show that the branches have the same late-return rate?

Reading software output

Reporting template

One proportion: [x] of [n] [units] ([ ]%) [had the outcome], 95% Wilson CI [[ ], [ ]]; this [differs / does not differ] from the reference value of [p0] ([exact binomial / score z] test, p = [ ]).

Two proportions: [Outcome] occurred in [x1] of [n1] ([ ]%) in [group 1] and [x2] of [n2] ([ ]%) in [group 2], a difference of [ ] percentage points, Newcombe 95% CI [[ ], [ ]], z = [ ], p = [ ] ([with / without] continuity correction), relative risk = [ ], 95% CI [[ ], [ ]].

Results checklist:

• The "yes" outcome and the unit of analysis are defined (one outcome per unit).
• Counts and denominators are reported for every group, not only percentages.
• The test variant is named: exact or z; with or without continuity correction; score or Wald SE.
• Expected counts were checked; exact or Fisher's test used when they are small.
• The CI method is named (Wilson, Clopper-Pearson, Newcombe) and the Wald interval is avoided for small n.
• At least one effect measure with a CI is reported (risk difference for decisions; RR or OR as appropriate).
• For A/B tests: planned sample size, randomisation check (sample ratio) and no optional stopping are reported.
• Odds ratios are not described as "times more likely" unless the outcome is rare.

Common mistakes and misconceptions

• Not knowing the continuity correction default. R's prop.test and SciPy's chi2_contingency apply Yates to 2x2 tables; statsmodels' proportions_ztest does not. In Worked example A this moves p from .040 to .052.
• Choosing the variant after seeing the p-value. Picking whichever of .040 and .052 suits your story is p-hacking. Fix the method in the plan.
• Using the Wald interval for small n. $\hat{p} \pm 1.96\,SE$ can extend below 0 or above 1 and covers the true value less often than 95% of the time. Use Wilson. Note that statsmodels' proportion_confint defaults to the Wald ("normal") method.
• "The two CIs overlap, so there is no significant difference." Wrong. In Worked example A the Wilson intervals overlap (0.141 to 0.227 and 0.088 to 0.162), yet the difference is significant (p = .040). Compare using the CI of the difference.
• Reading an odds ratio as a relative risk. OR = 1.61 does not mean "61% more likely" here; the relative risk is 1.50. The two are close only when the outcome is rare (under about 10%).
• Peeking and stopping early in A/B tests. Checking the p-value every day and stopping when it first drops below .05 inflates the false positive rate far above 5%. Fix n in advance or use a sequential design.
• Counting visits instead of visitors. If a visitor returns five times, their five outcomes are not independent. Analyse one outcome per randomised unit.
• Using a two-sample test for paired yes/no data. Before-and-after yes/no answers from the same people need McNemar's test.
• "p = 0.04 means a 4% chance that page B is no better." No: it is the probability of a difference at least this large if the pages truly had equal order rates.

Self-test questions

1. You run prop.test(c(54, 36), c(300, 300)) in R and proportions_ztest([54, 36], [300, 300]) in Python and get p = .052 and p = .040. Explain the difference and how to make them agree.
2. Show the relationship between the two-proportion z statistic and the 2x2 chi-square statistic. When does it hold?
3. Why is the Wilson interval preferred over the Wald interval for a proportion such as 3 out of 25?
4. In a study, 30% of the exposed and 20% of the unexposed have the outcome. Compute the risk difference, the relative risk and the odds ratio. Which one would you give a manager, and why?
5. Interleaving check: name the test. (a) 14 of 20 Brightwell Bakery loaves pass a quality check; target pass rate 90%. (b) 120 Lakemoor students answer "yes/no" to a question before and after a lecture. (c) Click rates on two randomised newsletter versions, 2,000 recipients each. (d) Exam pass rates across four tutorial groups.
6. Before an A/B test, why should you calculate the sample size, and which quantity drives it most?

My notes and next steps

Sources and further reading

Structure aligned with the NIST/SEMATECH e-Handbook of Statistical Methods and the reporting elements named in APA JARS-Quant; all wording, data and examples here are original.

• Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212.
• Clopper, C. J., and Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4), 404–413.
• Yates, F. (1934). Contingency tables involving small numbers and the χ² test. Supplement to the Journal of the Royal Statistical Society, 1(2), 217–235.
• Newcombe, R. G. (1998). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine, 17(8), 873–890.
• Agresti, A., and Coull, B. A. (1998). Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician, 52(2), 119–126.
• Brown, L. D., Cai, T. T., and DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science, 16(2), 101–133.
• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum.
• Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133.
• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.4 Does the proportion of defectives meet requirements?, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain.
• R Core Team. prop.test, binom.test, chisq.test, power.prop.test documentation, R 4.5.1. statsmodels documentation for proportions_ztest, proportion_confint and confint_proportions_2indep (0.15); SciPy documentation for binomtest, chi2_contingency, relative_risk and odds_ratio (1.18).
• Further reading (by name only): de Smith, M. J., Statistical Analysis Handbook (2024 edition, statsref.com); OpenIntro Statistics; Agresti, A., Categorical Data Analysis; Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments; American Psychological Association, Publication Manual (7th ed.) and JARS-Quant.

Answer key

Worked example B. (1) $\hat{p} = 27/40 = 0.675$; expected 20 yes and 20 no, so the z approximation is acceptable. (2) $\sqrt{0.25/40} = 0.0791$, $z = 0.175/0.0791 = 2.21$, $z^2 = 4.90$. (3) Exact .038, score z .027, Wald-SE z .018 (z = 2.36); $0.675 \times 0.325 = 0.219$ is smaller than 0.25, so the Wald SE is smaller and z larger. (4) Clopper-Pearson [0.509, 0.814], Wilson [0.520, 0.799], Wald [0.530, 0.820]; report Wilson (or Clopper-Pearson when an exact, conservative interval is required). (5) 27 of 40 students (67.5%) used the library weekly, 95% Wilson CI [52.0%, 79.9%], which is not consistent with the claimed 50% (exact binomial test, p = .038).

Exercise C. (1) $H_0: p_{Central} = p_{North}$, $H_1: p_{Central} \neq p_{North}$; pooled $\bar{p} = 27/230 = 0.117$, expected late returns 14.1 and 12.9, all expected counts well above 5. (2) $\hat{p}_1 = 0.150$, $\hat{p}_2 = 0.082$; $SE_0 = \sqrt{0.117 \times 0.883 \times (1/120 + 1/110)} = 0.0425$; $z = 0.0682/0.0425 = 1.60$, p = .109. (3) R uncorrected: X-squared = 2.575, p = .1086; Python: z = 1.605, p = .1086. R's default applies Yates' correction, which shrinks the chi-square and so gives a larger p. (4) RD = 0.068, Newcombe 95% CI [-0.017, 0.152]; RR = 1.83, 95% CI [0.86, 3.91]; OR = 1.98, 95% CI [0.85, 4.62]. (5) "Late returns occurred in 18 of 120 loans (15.0%) at Central and 9 of 110 (8.2%) at North, a difference of 6.8 percentage points, Newcombe 95% CI [-1.7, 15.2], z = 1.60, p = .109 (no continuity correction), relative risk = 1.83, 95% CI [0.86, 3.91]." No: the data are compatible with no difference but also with Central having almost four times North's rate. The study is too small to tell; a planned larger sample (or pooling several months) is needed.

Self-test.

1. R's prop.test applies Yates' continuity correction by default; proportions_ztest does not. Use prop.test(..., correct = FALSE) in R; both then give p = .040.
2. $z^2 = \chi^2$ for the 2x2 table when the z-test uses the pooled SE, the chi-square is Pearson's without continuity correction, and the z-test is two-sided.
3. With $\hat{p} = 0.12$ and n = 25 the Wald interval is $0.12 \pm 0.127$, which runs below 0 (statsmodels silently clips it at 0), and its actual coverage is well under 95%. The Wilson interval stays inside [0, 1], is asymmetric where it should be, and has coverage close to 95%.
4. RD = 0.10 (10 percentage points); RR = 0.30/0.20 = 1.5; OR = (0.30/0.70)/(0.20/0.80) = 0.4286/0.25 = 1.71. For a manager, the risk difference: "10 more cases per 100 people" is concrete; ratios hide the baseline.
5. (a) One-proportion test against 0.90; expected failures are 20 x 0.10 = 2, so use the exact binomial test. (b) McNemar's test (paired yes/no). (c) Two-proportion z-test (or chi-square without correction); with n = 2,000 per arm the correction hardly matters. (d) Chi-square test of independence on a 4x2 table (academic/stats-contingency-tables).
6. To have a good chance (power) of detecting the smallest effect worth acting on, and to fix the stopping point in advance so that peeking cannot inflate false positives. The minimum detectable difference drives it most: halving the difference roughly quadruples the required sample size.

Revision history