Confidence Intervals Concept Card Template (Statistics, R and Python)

Confidence Intervals Concept Card: [Your study question or course]

A concept card for confidence intervals: what "95% confidence" really means (the coverage of a procedure), how to build intervals for a mean (z and t), a proportion (Wald and Wilson) and a difference in means, how width depends on sample size and confidence level, how an interval links to a two-sided test, and a first look at the bootstrap. It is for students preparing for a statistics exam, master's and PhD students reporting estimates in a thesis, and analysts who must explain an interval to a stakeholder. It speeds up learning with a seeded coverage simulation you can watch, one fully worked example, two faded ones, and self-test questions with a separate answer key. The worked examples use fictional data; replace them with your own. Source basis: Neyman (1937), Wilson (1927), Welch (1947), Efron (1979), Agresti and Coull (1998), the NIST/SEMATECH e-Handbook, with all text and examples written fresh.

Document control

The idea in one paragraph

A confidence interval is a range of plausible values for an unknown population quantity, built from a sample by a recipe that has a known success rate. "95% confidence" describes the recipe, not one particular interval: if we repeated the study many times and built an interval each time, about 95% of those intervals would contain the true value. Any single interval either contains the truth or does not; we just do not know which. The interval's width tells you how precisely the data pin the quantity down, which a p-value alone never shows. Report an estimate with its interval as the main result, and treat a test decision as secondary.

Intuition

Think of throwing a ring at a peg you cannot see. The peg is the true mean μ. Each study throws one ring: the interval. A "95% ring" is wide enough that 95 throws out of 100 land around the peg. After a throw you cannot see whether this ring caught the peg, but you know the throwing method rarely misses. A bigger ring (99% confidence) misses less often but says less about where the peg is; a steadier hand (larger sample size n) lets you use a smaller ring with the same success rate.

The chart shows 20 intervals from 20 simulated studies (R, seed 7), each with n = 10 values drawn from a normal population with true mean 50 and SD 8 (generated with set.seed(7) and then, twenty times, t.test(rnorm(10, 50, 8))$conf.int). Nineteen intervals cover 50; interval 7, [53.07, 57.80], misses. That is what 95% coverage looks like in a small batch: about one miss in twenty, sometimes none, sometimes two or three.

Formal definition

Coverage. In words: before the data are collected, the random interval [L, U] has probability 1 − α of containing the fixed true value θ. After the data are in, the computed interval is fixed and the probability statement no longer applies to it.

$P\left(L(X) \le \theta \le U(X)\right) = 1 - \alpha$

General shape. Most textbook intervals are an estimate plus or minus a margin of error, where the margin is a critical value times the standard error SE:

$\text{estimate} \pm \text{critical value} \times SE$

Mean, population SD σ known (z-interval).

$\bar{x} \pm z_{1-\alpha/2}\,\frac{\sigma}{\sqrt{n}}$

Mean, σ unknown (t-interval). Replace σ with the sample SD s and the normal critical value with a t critical value; the t distribution has heavier tails to allow for the extra uncertainty in s.

$\bar{x} \pm t_{1-\alpha/2,\,n-1}\,\frac{s}{\sqrt{n}}$

Proportion, Wald interval. Simple but unreliable for small n or for proportions near 0 or 1.

$\hat{p} \pm z_{1-\alpha/2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

Proportion, Wilson interval (Wilson 1927). Obtained by inverting the score test; its centre is pulled towards 0.5 and it never leaves [0, 1].

$\frac{\hat{p} + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} \pm \frac{z}{1 + \frac{z^2}{n}}\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}$

Difference in two means, Welch (1947). No equal-variance assumption; df comes from the Welch-Satterthwaite formula.

$(\bar{x}_1 - \bar{x}_2) \pm t_{1-\alpha/2,\,\nu}\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}, \qquad \nu = \frac{\left(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}\right)^2}{\frac{(s_1^2/n_1)^2}{n_1 - 1} + \frac{(s_2^2/n_2)^2}{n_2 - 1}}$

Link to tests. A two-sided test of H0: θ = θ0 at level α rejects exactly when θ0 lies outside the (1 − α) confidence interval built from the same method. A CI is the set of null values the test would not reject.

Width: sample size and confidence level

Three things set the width of a t-interval: the spread s, the sample size n and the confidence level. Width shrinks with $\sqrt{n}$, so halving the width needs about four times the sample. Higher confidence needs a larger critical value: 1.645 for 90%, 1.960 for 95% and 2.576 for 99% on the normal scale. Small samples pay a further penalty through the t critical value:

For the Example A data (n = 10), the 90%, 95% and 99% intervals are [9.63, 12.57], [9.29, 12.91] and [8.49, 13.71] hours: more confidence, wider interval, same data.

Why the Wald interval fails

The Wald interval assumes the sampling distribution of $\hat{p}$ is normal and plugs $\hat{p}$ into the SE. Both steps break for small n or extreme proportions: the distribution is skewed, and when $\hat{p} = 0$ the estimated SE is 0, so 0 of 20 gives the absurd interval [0, 0]. Its actual coverage is often far below the nominal 95% and jumps erratically as n changes (Brown, Cai and DasGupta 2001). The Wilson interval keeps close to 95% coverage even in small samples and is the default in this library; the exact Clopper-Pearson interval guarantees at least 95% coverage but is conservative (wider than needed).

Bootstrap preview

When no formula is handy (a median, a ratio, a trimmed mean) or the sampling distribution is awkward, the bootstrap (Efron 1979) builds it from the data: resample the n observations with replacement many times, compute the statistic each time, and take the 2.5th and 97.5th percentiles of those values as a percentile interval. It assumes the sample represents the population well, so it is weak for very small n. Full treatment, including better variants such as BCa, is in stats-resampling-methods.

Worked example A: fully solved

Question. How many hours a week do Lakemoor University (fictional) second-year students study outside class? Ten students kept a diary for one week.

Step 1: choose the interval. One mean, population SD unknown, so a t-interval with df = 9. Assumptions: students are independent; hours are roughly symmetric with no extreme outliers (true here).

Step 2: compute. Outputs on this card are trimmed to the key lines (for example R's attr(,"conf.level") and data: lines are dropped).

h <- c(12.5, 8.0, 15.0, 10.5, 9.0, 14.0, 11.5, 7.5, 13.0, 10.0)
n <- length(h); m <- mean(h); s <- sd(h); se <- s / sqrt(n)
tc <- qt(0.975, df = n - 1)
round(c(n = n, mean = m, sd = s, se = se, tcrit = tc,
        lo = m - tc * se, hi = m + tc * se), 3)
t.test(h)$conf.int                      # same interval in one line

     n   mean     sd     se  tcrit     lo     hi
10.000 11.100  2.536  0.802  2.262  9.286 12.914
[1]  9.285568 12.914432
attr(,"conf.level")
[1] 0.95

import numpy as np
from scipy import stats

h = np.array([12.5, 8.0, 15.0, 10.5, 9.0, 14.0, 11.5, 7.5, 13.0, 10.0])
n, m, se = h.size, h.mean(), stats.sem(h)
print(round(m, 3), round(se, 3))
lo, hi = stats.t.interval(0.95, df=n - 1, loc=m, scale=se)
print(float(lo), float(hi))
ci = stats.ttest_1samp(h, 0).confidence_interval(0.95)
print(float(ci.low), float(ci.high))

11.1 0.802
9.285567562904356 12.914432437095643
9.285567562904356 12.914432437095643

Step 3: by hand. $\bar{x}$ = 11.10, s = 2.536, SE = 2.536 / √10 = 0.802. With $t_{0.975, 9}$ = 2.262, the margin of error is 2.262 × 0.802 = 1.814, so the interval is 11.10 ± 1.81 = [9.29, 12.91] hours.

Step 4: z versus t. Using 1.960 instead of 2.262 would give [9.53, 12.67]: too narrow, because it ignores the uncertainty in s. With n = 10 that matters; with n = 100 the difference is negligible.

Step 5: interpret. We estimate that these students study 11.1 hours a week on average, and values from about 9.3 to 12.9 hours are compatible with the data at the 95% level. The method that produced this range captures the true mean in 95% of repeated samples. The interval excludes, for example, a claimed 8 hours, so a two-sided test of H0: μ = 8 would reject at α = 0.05.

Report: Students reported studying M = 11.10 hours per week (SD = 2.54, n = 10), 95% CI [9.29, 12.91].

Wald versus Wilson in practice

Three of 25 sampled Lakemoor students use the night bus ($\hat{p}$ = 0.12).

x <- 3; n <- 25; ph <- x / n
round(ph + c(-1, 1) * qnorm(0.975) * sqrt(ph * (1 - ph) / n), 4)   # Wald by hand
round(prop.test(x, n, correct = FALSE)$conf.int, 4)      # Wilson
round(binom.test(x, n)$conf.int, 4)                      # Clopper-Pearson
round(prop.test(0, 20, correct = FALSE)$conf.int, 4)     # Wilson for 0 of 20

[1] -0.0074  0.2474
[1] 0.0417 0.2996
[1] 0.0255 0.3122
[1] 0.0000 0.1611

import numpy as np
from statsmodels.stats.proportion import proportion_confint
for meth in ["normal", "wilson", "beta"]:      # normal = Wald, beta = Clopper-Pearson
    print(meth, np.round(proportion_confint(3, 25, alpha=0.05, method=meth), 4))
print(np.round(proportion_confint(0, 20, method="normal"), 4),
      np.round(proportion_confint(0, 20, method="wilson"), 4))

normal [0.     0.2474]
wilson [0.0417 0.2996]
beta [0.0255 0.3122]
[0. 0.] [0.     0.1611]

The Wald interval computed by hand in R goes below zero (−0.0074), an impossible proportion; statsmodels proportion_confint clips its "normal" (Wald) limits to [0, 1] (documented behaviour), so it prints 0, which hides the problem rather than fixing it. Wilson gives [0.042, 0.300] and stays inside [0, 1]. For 0 successes out of 20, Wald collapses to [0, 0] while Wilson gives [0, 0.161]: honest uncertainty.

Worked example B: half solved

Question. Do students in Lakemoor's North Hall (n = 8) study more hours than those in South Hall (n = 7)?

a <- c(11.0, 13.5, 9.5, 12.0, 14.5, 10.0, 12.5, 13.0)
b <- c(9.0, 10.5, 8.0, 11.0, 9.5, 7.5, 10.0)
t.test(a, b)                              # Welch is R's default
round(t.test(a, b, var.equal = TRUE)$conf.int, 3)   # pooled, for comparison

	Welch Two Sample t-test
t = 3.3847, df = 12.7, p-value = 0.005033
95 percent confidence interval:
 0.9519211 4.3337932
sample estimates:
mean of x mean of y
12.000000  9.357143
[1] 0.920 4.365

import numpy as np
from scipy import stats
a = np.array([11.0, 13.5, 9.5, 12.0, 14.5, 10.0, 12.5, 13.0])
b = np.array([9.0, 10.5, 8.0, 11.0, 9.5, 7.5, 10.0])
r = stats.ttest_ind(a, b, equal_var=False)     # scipy default is pooled: set equal_var=False
ci = r.confidence_interval(0.95)
print(float(r.statistic), float(r.pvalue), float(r.df))
print(float(ci.low), float(ci.high))
ci_pooled = stats.ttest_ind(a, b).confidence_interval(0.95)
print(float(ci_pooled.low), float(ci_pooled.high))

3.3846812177229766 0.005033278891070464 12.700140813755281
0.9519210804350728 4.333793205279212
0.9202523026622391 4.365461983052046

Note the defaults: R's t.test uses Welch unless told otherwise, while scipy's ttest_ind uses the pooled-variance test unless you set equal_var=False. Always state which one you ran.

1. Point estimate of the difference: 12.000 − 9.357 = [ ] hours.
2. SE of the difference: $\sqrt{1.732^2/8 + 1.282^2/7}$ = [ ] (SDs 1.732 and 1.282).
3. Critical value $t_{0.975,\,12.7}$ = 2.166; margin of error = [ ].
4. 95% Welch CI: [ ] to [ ] hours.
5. Does the interval contain 0? [ ]. What does that imply for a two-sided Welch test at α = 0.05? [ ].
6. Report sentence: "North Hall students studied [ ] hours more per week, 95% CI [ ], t([ ]) = 3.38, p = .005."

Exercise C: unsolved

Brightwell Bakery (fictional) times how long 16 croissant batches take to proof. The mean is 72.4 minutes and the sample SD is 6.0 minutes.

1. Compute the SE and the 95% t-interval (use $t_{0.975, 15}$ = 2.131).
2. Compute the 99% t-interval ($t_{0.995, 15}$ = 2.947). Which is wider and why?
3. How many batches would roughly halve the width of the 95% interval?
4. A supplier claims proofing takes 80 minutes on average. What does the 95% interval say about that claim, without running a separate test?
5. Rewrite this sentence correctly: "There is a 95% probability that the true mean lies in my interval."

Simulation you can run

Coverage of the t-interval. Draw 10,000 samples of n = 10 from a normal population with mean 50 and SD 8, build a 95% t-interval for each, and count how many contain 50. Wald versus Wilson. Draw 10,000 binomial counts with n = 20 and true π = 0.10 and check each interval.

set.seed(2026)
cover <- replicate(10000, {
  y <- rnorm(10, 50, 8); ci <- t.test(y)$conf.int; ci[1] <= 50 && 50 <= ci[2]
})
mean(cover)

set.seed(2026)
k <- rbinom(10000, 20, 0.1)
wald <- sapply(k, function(x) { p <- x / 20
  ci <- p + c(-1, 1) * qnorm(0.975) * sqrt(p * (1 - p) / 20); ci[1] <= .1 && .1 <= ci[2] })
wilson <- sapply(k, function(x) { ci <- prop.test(x, 20, correct = FALSE)$conf.int
  ci[1] <= .1 && .1 <= ci[2] })
c(wald = mean(wald), wilson = mean(wilson))

h <- c(12.5, 8.0, 15.0, 10.5, 9.0, 14.0, 11.5, 7.5, 13.0, 10.0)   # Example A
set.seed(59)                               # bootstrap preview on Example A
bm <- replicate(5000, mean(sample(h, replace = TRUE)))
quantile(bm, c(0.025, 0.975))

[1] 0.9468
  wald wilson
0.8743 0.9572
 2.5% 97.5%
 9.60 12.65

import numpy as np
from scipy import stats
from statsmodels.stats.proportion import proportion_confint

rng = np.random.default_rng(2026)
y = rng.normal(50, 8, size=(10000, 10))
m, se = y.mean(1), y.std(1, ddof=1) / np.sqrt(10)
tc = stats.t.ppf(0.975, 9)
print(((m - tc * se <= 50) & (50 <= m + tc * se)).mean())

k = rng.binomial(20, 0.1, size=10000)
lo, hi = proportion_confint(k, 20, method="normal"); print("wald", ((lo <= .1) & (.1 <= hi)).mean())
lo, hi = proportion_confint(k, 20, method="wilson"); print("wilson", ((lo <= .1) & (.1 <= hi)).mean())

h = np.array([12.5, 8.0, 15.0, 10.5, 9.0, 14.0, 11.5, 7.5, 13.0, 10.0])
bs = stats.bootstrap((h,), np.mean, n_resamples=5000, method="percentile",
                     rng=np.random.default_rng(59))
print(float(bs.confidence_interval.low), float(bs.confidence_interval.high))

0.9497
wald 0.8729
wilson 0.957
9.55 12.6

Read the results. The t-interval covers the true mean 94.7% (R) and 95.0% (Python) of the time: on target, given a Monte Carlo SE of about 0.002. The Wald interval for π = 0.10 and n = 20 covers only about 87% of the time in both languages, far short of its 95% promise, while Wilson covers about 96%. The bootstrap percentile interval for the Example A mean, [9.60, 12.65] in R and [9.55, 12.60] in Python, is close to the t-interval [9.29, 12.91] but a little narrower, a known tendency of the percentile bootstrap in small samples. R and Python use different random number generators, so their simulated numbers differ slightly even with the same seed; both are correct.

Misconceptions

Connections to other concepts

• Sampling distributions (stats-sampling-distributions): the SE in every formula is the SD of a sampling distribution; the CLT justifies the normal and t shapes.
• Hypothesis testing (stats-hypothesis-testing): a 95% CI contains exactly the null values a two-sided test at α = 0.05 would not reject.
• Effect sizes (stats-effect-sizes): report every effect size with its interval.
• Proportion tests (stats-proportion-tests): Wilson and Newcombe intervals for one and two proportions in full.
• Power and sample size (stats-power-sample-size): plan n for a target margin of error, not only for a target power.
• Resampling (stats-resampling-methods): bootstrap intervals for statistics without formulas.
• Bayesian inference (stats-bayesian-inference): credible intervals answer the question many people think a CI answers.

Self-test

1. Complete the sentence correctly: "95% confidence means that ..."
2. For n = 10, why is the t critical value (2.262) larger than the normal one (1.960), and what happens to the difference as n grows?
3. Give two reasons the Wald interval for a proportion can fail, and name a better default.
4. You want to cut a margin of error from 4 points to 2 points. Roughly how does the required n change?
5. A 95% CI for a difference in means is [0.95, 4.33]. What is the two-sided p-value: above or below 0.05? Explain without computing.
6. When would you choose a bootstrap interval, and when is it weak?

My notes and next steps

My own explanation of "95% confidence" for a friend: [write it here without looking].

Sources and further reading

Structure aligned with the GAISE College Report (ASA 2016) emphasis on estimation and simulation-based understanding; based on the original method papers below. All wording and examples are original.

• Agresti, A., and Coull, B. A. (1998). Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician, 52(2), 119–126.
• Brown, L. D., Cai, T. T., and DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science, 16(2), 101–133.
• Clopper, C. J., and Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4), 404–413.
• Cumming, G., and Maillardet, R. (2006). Confidence intervals and replication: where will the next mean fall? Psychological Methods, 11(3), 217–227.
• Cumming, G. (2014). The new statistics: why and how. Psychological Science, 25(1), 7–29.
• Efron, B. (1979). Bootstrap methods: another look at the jackknife. The Annals of Statistics, 7(1), 1–26.
• Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London, Series A, 236(767), 333–380.
• Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1–2), 28–35.
• Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212.
• GAISE College Report ASA Revision Committee (2016). Guidelines for Assessment and Instruction in Statistics Education: College Report 2016. American Statistical Association.
• NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.5.2 Confidence Limits for the Mean, and section 7.2.4.1 Confidence intervals (for a proportion), https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain.
• Further reading (by name only): de Smith, M. J. (2024). Statistical Analysis Handbook (2024 edition, StatsRef); OpenIntro Statistics (CC BY-SA); APA Publication Manual (7th ed.) and APA JARS-Quant for reporting.

Answer key

Example B.

1. 2.643 hours.
2. SE = $\sqrt{3.000/8 + 1.643/7}$ = 0.781.
3. Margin = 2.166 × 0.781 = 1.691.
4. [0.95, 4.33] hours (pooled version: [0.92, 4.37]).
5. No, 0 is outside the interval, so the two-sided Welch test rejects H0 of equal means at α = 0.05 (p = .005).
6. "North Hall students studied 2.64 hours more per week, 95% CI [0.95, 4.33], t(12.7) = 3.38, p = .005."

Exercise C.

1. SE = 6.0 / √16 = 1.5; margin = 2.131 × 1.5 = 3.197; 95% CI [69.20, 75.60] minutes.
2. Margin = 2.947 × 1.5 = 4.42; 99% CI [67.98, 76.82]. Wider, because a higher success rate needs a larger critical value.
3. About four times as many, so about 64 batches (width scales with $1/\sqrt{n}$; the t critical value also drops a little).
4. 80 minutes lies outside [69.20, 75.60], so the data are not compatible with the claim at the 95% level; a two-sided t-test of H0: μ = 80 would reject at α = 0.05.
5. "If we repeated this study many times, 95% of the intervals built this way would contain the true mean; this interval is one of them."

Self-test.

1. "... the method produces intervals that contain the true value in 95% of repeated samples."
2. The t distribution allows for the extra uncertainty from estimating σ with s; its heavier tails need a larger cut-off. As n grows, t approaches the normal and the critical value approaches 1.960.
3. Its sampling distribution is skewed for small n or extreme proportions, and the plug-in SE is 0 when $\hat{p}$ is 0 or 1 (it can also produce limits outside [0, 1]). Use Wilson.
4. Width scales with $1/\sqrt{n}$, so halving the margin needs about four times the sample size.
5. Below 0.05, because 0 is outside the 95% interval for the difference.
6. Use it for statistics without a reliable formula (median, ratio, trimmed mean) or for skewed sampling distributions; it is weak for very small samples because the sample may poorly represent the population.

Revision history