Hypothesis Testing Concept Card Template (Statistics, R and Python)

Hypothesis Testing Concept Card: [Your study question or course]

A concept card for the logic of significance testing: null and alternative hypotheses, test statistics, p-values, the significance level α, Type I and Type II errors, power, one- and two-sided tests, statistical versus practical significance, equivalence testing and multiple testing (Bonferroni, Holm, Benjamini-Hochberg). It is for students preparing for a statistics exam, master's and PhD students about to test thesis hypotheses, and analysts who must explain a p-value to a colleague. It speeds up learning by putting the decisions in order, showing one fully worked example and two faded ones, and ending with self-test questions whose answers sit in a separate key. The worked examples use fictional data; replace them with your own. Source basis: Fisher (1925), Neyman and Pearson (1933), the ASA Statement on Statistical Significance and P-Values (2016), the NIST/SEMATECH e-Handbook, with all text and examples written fresh.

Document control

The idea in one paragraph

A hypothesis test asks one narrow question: if a simple reference model (the null hypothesis, H0) were true, how surprising would data like ours be? We summarise the data in a test statistic, work out how that statistic would vary if H0 were true, and report the p-value: the probability, computed under H0, of a statistic at least as extreme as the one observed. A small p-value says the data sit awkwardly with H0; it does not say how likely H0 is, and it does not say how large or important the effect is. A significance test is therefore one input to a judgement, not the judgement itself.

Intuition

Think of a smoke alarm. The null hypothesis is "there is no fire". The alarm (the test) is tuned so that it rarely sounds when there is no fire: that tuning is the significance level α, the false-alarm rate you accept. When it does sound, you have reason to act, but a false alarm is still possible (a Type I error). When it stays silent, a small fire may still be burning (a Type II error). A more sensitive alarm (more power) catches more real fires; you buy sensitivity with better sensors, which in statistics means a larger sample size n, less noise, or a larger true effect.

Two historical traditions are blended in most textbooks. Fisher (1925) treated the p-value as a graded measure of evidence against a single null hypothesis; he called 0.05 a convenient reference point but did not treat it as a rigid rule, and he used no formal alternative. Neyman and Pearson (1933) instead framed testing as a decision rule between two hypotheses, fixing α in advance and choosing the test that maximises power; in their framework only the decision (reject or not) matters, and the error rates are long-run properties of the procedure. Modern practice reports the exact p-value (Fisher) and also compares it with a pre-chosen α (Neyman-Pearson). Knowing that the hybrid exists explains why textbooks sometimes seem to contradict each other.

Formal definition

Hypotheses. The null hypothesis states a specific value or relation, usually "no effect"; the alternative says what you expect otherwise. For a population mean μ and a reference value μ0:

$H_0: \mu = \mu_0 \qquad H_1: \mu \neq \mu_0 \;\text{(two-sided)} \quad\text{or}\quad H_1: \mu < \mu_0 \;\text{(one-sided)}$

Test statistic. A test statistic measures how far the estimate is from the null value in units of its standard error SE. For a one-sample t-test:

$t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}}, \qquad df = n - 1$

p-value. In words: the probability, assuming H0 and all the model assumptions are true, of getting a test statistic at least as extreme as the observed value $t_{obs}$. For a two-sided test:

$p = P\left(|T| \ge |t_{obs}| \;\middle|\; H_0\right)$

Decision rule. Reject H0 when $p \le \alpha$. Equivalently, reject when $|t_{obs}|$ exceeds the critical value $t_{1-\alpha/2,\,df}$.

Errors and power.

$\alpha = P(\text{reject } H_0 \mid H_0 \text{ true}), \qquad \beta = P(\text{fail to reject } H_0 \mid H_1 \text{ true}), \qquad \text{power} = 1 - \beta$

One-sided or two-sided

A two-sided test looks for a difference in either direction; it is the default in this library and in most journals. A one-sided test puts the whole of α in one tail, so it has more power for an effect in the predicted direction and none at all for an effect in the other direction. Use it only when the direction is fixed before the data are seen and an effect in the other direction would lead to exactly the same action as no effect (for example, a regulator only cares whether loaves are underweight). Switching to one-sided after seeing the data halves the p-value dishonestly. For a symmetric statistic, the one-sided p-value in the observed direction is half the two-sided p-value.

Statistical and practical significance

A p-value mixes two things: the size of the effect and the amount of information (mostly n). With a huge sample, a trivial difference such as 0.1 g in a 500 g loaf can give p < .001; with a tiny sample, a large and important difference can give p = 0.20. Always report an effect size (here Cohen's d) and a confidence interval, and judge them against a smallest effect size of interest that you set from subject knowledge, such as a regulatory tolerance, a minimal clinically important difference, or a cost threshold.

Equivalence testing (TOST)

A non-significant result does not show that an effect is absent. To claim "no meaningful difference", define an equivalence margin $[-\Delta, \Delta]$ in advance and run two one-sided tests (TOST, Schuirmann 1987): test $H_{01}: \mu - \mu_0 \le -\Delta$ and $H_{02}: \mu - \mu_0 \ge \Delta$. If both are rejected at level α, the effect lies inside the margin. Equivalently, the 90% confidence interval (for α = 0.05) lies entirely within the margin. Lakens (2017) gives an accessible introduction.

Multiple testing

Each test at α = 0.05 carries a 5% false-alarm risk when its null is true. Run m independent tests on true nulls and the chance of at least one false positive, the family-wise error rate (FWER), is $1 - (1-\alpha)^m$: about 0.40 for m = 10. Three common corrections:

• Bonferroni controls FWER by testing each hypothesis at $\alpha/m$, or equivalently multiplying each p-value by m (capped at 1); Dunn (1961) popularised it for comparing means. Simple and valid under any dependence, but conservative.
• Holm (1979) is a step-down version that is never less powerful than Bonferroni and still controls FWER. Sort p-values $p_{(1)} \le \dots \le p_{(m)}$; compare $p_{(k)}$ with $\alpha/(m-k+1)$ and stop at the first failure.
• Benjamini-Hochberg (1995) controls the false discovery rate (FDR), the expected proportion of false positives among the rejected hypotheses. Find the largest k with $p_{(k)} \le \frac{k}{m}\alpha$ and reject hypotheses $1, \dots, k$. It is more powerful than FWER methods and suits screening many outcomes, as long as the tests are independent or positively dependent.

$\text{FWER} = P(\text{at least one false rejection}), \qquad \text{FDR} = E\left[\frac{V}{\max(R, 1)}\right]$

where V is the number of false rejections and R the total number of rejections.

The ASA statement in six sentences

The American Statistical Association's 2016 statement (Wasserstein and Lazar 2016) set out six principles. In this card's own words:

1. A p-value tells you how poorly the data fit one particular model, the null model together with all its assumptions.
2. It is not the probability that the hypothesis is true, and it is not the probability that the result is "due to chance alone".
3. Crossing a threshold such as 0.05 should never by itself decide a scientific conclusion, a business choice or a policy.
4. Sound inference needs full reporting: every analysis run, every p-value computed, and how the analyses were chosen.
5. A p-value says nothing about how large or how important an effect is.
6. On its own, a p-value is a weak summary of evidence; pair it with estimates, intervals, design information and other approaches.

Worked example A: fully solved

Question. Brightwell Bakery (fictional) labels its sourdough loaf as 500 g. A quality assistant weighs 12 loaves from one morning batch. Is the mean weight different from 500 g?

Step 1: hypotheses. H0: μ = 500 g; H1: μ ≠ 500 g (two-sided, because overweight loaves cost money and underweight loaves break the label promise). α = 0.05.

Step 2: assumptions. Loaves are independent draws from the batch; weights are roughly normal (a normal Q-Q plot of 12 points shows no strong skew or outlier). With n = 12 the t-test is reasonably robust to mild non-normality.

Step 3: compute. The outputs below are trimmed to the key lines (R's data: and sample estimates: lines are dropped).

w <- c(497.2, 501.5, 494.8, 498.9, 496.1, 499.4, 495.7, 500.2,
       497.8, 493.9, 498.5, 496.6)
round(c(n = length(w), mean = mean(w), sd = sd(w), se = sd(w) / sqrt(length(w))), 3)
t.test(w, mu = 500)                          # two-sided by default
t.test(w, mu = 500, alternative = "less")$p.value

      n    mean      sd      se
 12.000 497.550   2.262   0.653

	One Sample t-test
t = -3.7524, df = 11, p-value = 0.003196
alternative hypothesis: true mean is not equal to 500
95 percent confidence interval:
 496.113 498.987
mean of x
   497.55
[1] 0.001598153

import numpy as np
from scipy import stats

w = np.array([497.2, 501.5, 494.8, 498.9, 496.1, 499.4, 495.7, 500.2,
              497.8, 493.9, 498.5, 496.6])
res = stats.ttest_1samp(w, popmean=500)          # two-sided by default
print(float(res.statistic), float(res.pvalue), res.df)
ci = res.confidence_interval(0.95)
print(float(ci.low), float(ci.high))
print(stats.ttest_1samp(w, 500, alternative="less").pvalue)

-3.7524472199642314 0.003196306989366216 11
496.11296082899315 498.98703917100687
0.001598153494683108

Step 4: by hand. The mean is 497.55 g and s = 2.262 g, so SE = 2.262 / √12 = 0.653 g. Then t = (497.55 − 500) / 0.653 = −3.75 with df = 11. The two-sided critical value $t_{0.975, 11}$ is about 2.20; since |−3.75| > 2.20, we reject H0. The one-sided p-value (0.0016) is exactly half of the two-sided one (0.0032), as expected for a symmetric statistic.

Step 5: effect size. Cohen's d = (497.55 − 500) / 2.262 = −1.08: the batch mean is about one standard deviation below the label. The 95% CI for the mean, [496.11, 498.99] g, excludes 500, matching the test.

Step 6: interpret and report. The batch averaged 2.45 g below the label. Whether that matters is a business and legal question (for instance, whether the shortfall breaks a tolerance rule), not a statistical one.

Report: Mean loaf weight (M = 497.55 g, SD = 2.26, n = 12) was lower than the labelled 500 g, t(11) = −3.75, p = .003, d = −1.08, 95% CI [496.11, 498.99].

Power check. If the true shortfall were 2 g with SD 2.5 g, a two-sided test with n = 12 would detect it with power 0.71; about 15 loaves (14.3 rounded up) would give 80% power.

power.t.test(n = 12, delta = 2, sd = 2.5, sig.level = 0.05, type = "one.sample")$power
power.t.test(power = 0.8, delta = 2, sd = 2.5, sig.level = 0.05, type = "one.sample")$n

[1] 0.713657
[1] 14.3028

from statsmodels.stats.power import TTestPower
print(TTestPower().power(effect_size=2/2.5, nobs=12, alpha=0.05))
print(TTestPower().solve_power(effect_size=2/2.5, power=0.8, alpha=0.05))

0.7136600656001074
14.302764558129445

Multiple testing: worked p-value adjustment

The Lakemoor University (fictional) teaching office compares eight courses' pass rates with last year's and gets eight raw p-values. How many differences survive each correction at α = 0.05?

p <- c(0.001, 0.008, 0.012, 0.021, 0.034, 0.047, 0.210, 0.640)
tab <- data.frame(p = p,
                  bonferroni = p.adjust(p, "bonferroni"),
                  holm       = p.adjust(p, "holm"),
                  BH         = p.adjust(p, "BH"))
round(tab, 4)
colSums(tab < 0.05)

      p bonferroni  holm     BH
1 0.001      0.008 0.008 0.0080
2 0.008      0.064 0.056 0.0320
3 0.012      0.096 0.072 0.0320
4 0.021      0.168 0.105 0.0420
5 0.034      0.272 0.136 0.0544
6 0.047      0.376 0.141 0.0627
7 0.210      1.000 0.420 0.2400
8 0.640      1.000 0.640 0.6400
         p bonferroni       holm         BH
         6          1          1          4

import numpy as np
from statsmodels.stats.multitest import multipletests

p = np.array([0.001, 0.008, 0.012, 0.021, 0.034, 0.047, 0.210, 0.640])
for m in ["bonferroni", "holm", "fdr_bh"]:
    rej, padj, _, _ = multipletests(p, alpha=0.05, method=m)
    print(m, np.round(padj, 4), rej.sum())

bonferroni [0.008 0.064 0.096 0.168 0.272 0.376 1.    1.   ] 1
holm [0.008 0.056 0.072 0.105 0.136 0.141 0.42  0.64 ] 1
fdr_bh [0.008  0.032  0.032  0.042  0.0544 0.0627 0.24   0.64  ] 4

R and Python agree to four decimals. Unadjusted, six courses look "significant". Bonferroni and Holm keep only the first (FWER control); Benjamini-Hochberg keeps four (FDR control). By hand for BH: with m = 8 the thresholds $\frac{k}{8} \times 0.05$ are 0.00625, 0.0125, 0.01875, 0.025, 0.03125, 0.0375, ... The largest k with $p_{(k)}$ below its threshold is k = 4 (0.021 ≤ 0.025), so hypotheses 1 to 4 are rejected. For Holm, the second p-value 0.008 is compared with 0.05/7 = 0.00714 and fails, so the procedure stops after the first.

Worked example B: half solved

Question. Harbour City Library (fictional) claims that 60% of users are satisfied with the self-checkout kiosks. In a survey of 40 users, 31 say they are satisfied. Is the satisfaction rate different from 60%?

1. Hypotheses: H0: π = 0.60; H1: π [ ] 0.60 (two-sided).
2. Sample proportion: $\hat{\pi}$ = 31 / 40 = [ ].
3. Test: exact binomial test, because the outcome is binary and n is small enough to compute exactly.

binom.test(31, 40, p = 0.6)

	Exact binomial test
number of successes = 31, number of trials = 40, p-value = 0.02391
alternative hypothesis: true probability of success is not equal to 0.6
95 percent confidence interval:
 0.6154883 0.8916034
sample estimates:
probability of success
                 0.775

from scipy import stats
print(stats.binomtest(31, 40, p=0.6))

BinomTestResult(k=31, n=40, alternative='two-sided', statistic=0.775, pvalue=0.023914394414508704)

4. Decision at α = 0.05: p = [ ], so [reject / do not reject] H0.
5. Does the 95% CI contain 0.60? [ ]. Is that consistent with step 4? [ ].
6. Report sentence: "[ ] of 40 users (77.5%) were satisfied, higher than the claimed 60%, exact binomial p = [ ], 95% CI [ ]."
7. Practical significance: is a difference of [ ] percentage points important for the library? [Your judgement].

Exercise C: unsolved

A Lakemoor University lab runs a two-sided independent-samples test and reports t = 2.10 with df = 24.

1. Find the two-sided p-value and the critical value at α = 0.05.
2. Decide at α = 0.05 and at α = 0.01.
3. The researcher had pre-registered a one-sided H1 in the observed direction. What is the one-sided p-value?
4. The same lab ran 20 such tests in the study. Under Bonferroni, what per-test threshold applies, and does this result survive?
5. Write one sentence explaining the result to a non-statistician without saying "probability that H0 is true".

Simulation you can run

Claim to see with your own eyes: when H0 is true and the test's assumptions hold, p-values are uniformly distributed between 0 and 1. That is why a test at α = 0.05 rejects a true H0 about 5% of the time: 5% of a uniform distribution lies below 0.05. Under a real effect, the distribution piles up near 0, and the share below 0.05 is the power.

set.seed(2026)
pv <- replicate(10000, t.test(rnorm(15, mean = 50, sd = 10), mu = 50)$p.value)
mean(pv < 0.05)                               # Type I error rate
table(cut(pv, breaks = seq(0, 1, 0.1)))       # ten equal bins
set.seed(2026)
pv1 <- replicate(10000, t.test(rnorm(15, mean = 55, sd = 10), mu = 50)$p.value)
mean(pv1 < 0.05)                              # power for a 5-point effect

[1] 0.0543
  (0,0.1] (0.1,0.2] (0.2,0.3] (0.3,0.4] (0.4,0.5] (0.5,0.6] (0.6,0.7] (0.7,0.8]
     1055      1000       977      1052      1039       957      1010       984
(0.8,0.9]   (0.9,1]
      949       977
[1] 0.4371

import numpy as np
from scipy import stats

rng = np.random.default_rng(2026)
x = rng.normal(50, 10, size=(10000, 15))
pv = stats.ttest_1samp(x, 50, axis=1).pvalue
print((pv < 0.05).mean())
print(np.histogram(pv, bins=np.linspace(0, 1, 11))[0])

0.0498
[ 994 1015  986 1028  939 1009 1016 1017 1009  987]

R and Python use different random number generators, so the same seed gives different draws: R found a rejection rate of 0.0543 and Python 0.0498. Both are within Monte Carlo error of 0.05: the simulation standard error is $\sqrt{0.05 \times 0.95 / 10000} \approx 0.0022$, so anything between about 0.0457 and 0.0543 is expected. Each of the ten bins holds roughly 1,000 of the 10,000 p-values. With a true shift of 5 points (d = 0.5) and n = 15, only 44% of the simulated studies reached p < 0.05: a single non-significant study says little when power is that low.

The chart shows the R counts per bin under H0. A flat profile is the signature of a well-calibrated test.

Misconceptions

Connections to other concepts

• Sampling distributions (stats-sampling-distributions): the null distribution of the test statistic is a sampling distribution computed under H0.
• Confidence intervals (stats-confidence-intervals): a two-sided test at level α rejects H0: μ = μ0 exactly when the (1 − α) CI excludes μ0. The interval carries more information than the decision.
• Effect sizes (stats-effect-sizes): the answer to "how much?", which a p-value cannot give.
• Power and sample size (stats-power-sample-size): plan n so that an effect worth detecting is detected with high probability, usually 80% or 90%.
• Bayesian inference (stats-bayesian-inference): answers "how probable is the hypothesis, given the data?", which a p-value does not.
• Resampling (stats-resampling-methods): permutation tests build the null distribution by reshuffling instead of relying on a formula.
• Analysis plans (stats-analysis-plan, academic/pre-analysis-plan): fix hypotheses, α, sidedness and multiplicity corrections before the data arrive.

Self-test

1. Define the p-value in one sentence that includes the words "assuming" and "at least as extreme".
2. A study has power 0.80 at α = 0.05. What are the probabilities of a Type I and a Type II error, and under which truth does each apply?
3. Why is a one-sided p-value half the two-sided one for a t-test, and when is a one-sided test acceptable?
4. Twelve hypotheses are tested. What is the Bonferroni per-test threshold at α = 0.05, and why is Holm preferred over Bonferroni?
5. Explain the difference between controlling the FWER and controlling the FDR, and give one setting where each is appropriate.
6. A huge study finds p < .001 for a 0.1 g difference in loaf weight. What should the report say about it?

My notes and next steps

My own one-sentence definition of a p-value: [write it here without looking, then compare with the formal definition].

Sources and further reading

Structure aligned with the GAISE College Report (ASA 2016) recommendations to stress conceptual understanding and with the ASA Statement on Statistical Significance and P-Values; based on the original method papers listed below. All wording and examples are original.

• Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1), 289–300.
• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates.
• Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293), 52–64.
• Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd.
• Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., and Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350.
• Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
• Lakens, D. (2017). Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362.
• Neyman, J., and Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A, 231, 289–337.
• Schuirmann, D. J. (1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15(6), 657–680.
• Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133.
• GAISE College Report ASA Revision Committee (2016). Guidelines for Assessment and Instruction in Statistics Education (GAISE) College Report 2016. American Statistical Association.
• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.1.3 What are statistical tests?, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain.
• Further reading: de Smith, M. J. (2024). Statistical Analysis Handbook, 2024 edition. StatsRef, https://www.statsref.com/; OpenIntro Statistics (CC BY-SA); APA Publication Manual (7th ed.) and APA JARS-Quant for reporting.

Answer key

Example B.

1. H1: π ≠ 0.60.
2. $\hat{\pi}$ = 0.775.
3. p = 0.024 (R 0.02391, Python 0.0239), so reject H0 at α = 0.05.
4. The 95% CI [0.615, 0.892] does not contain 0.60, consistent with rejecting H0 at α = 0.05. (The exact Clopper-Pearson interval and the exact test can disagree in borderline cases; here they agree.)
5. "31 of 40 users (77.5%) were satisfied, higher than the claimed 60%, exact binomial p = .024, 95% CI [0.62, 0.89]."
6. A difference of 17.5 percentage points; importance is a judgement for the library, but the CI shows the true rate could be as low as about 62%, only slightly above the claim.

Exercise C.

1. Two-sided p = 0.046 (R 2*pt(2.10, 24, lower.tail = FALSE) gives 0.04642; scipy 2*stats.t.sf(2.10, 24) gives 0.04642). Critical value $t_{0.975, 24}$ = 2.064.
2. At α = 0.05: reject, since 2.10 > 2.064 and 0.046 < 0.05. At α = 0.01: do not reject.
3. One-sided p = 0.023, valid only because the direction was pre-registered.
4. Threshold 0.05 / 20 = 0.0025; 0.046 does not survive.
5. Example: "If there were truly no difference between the groups, a difference at least as large as the one we saw (in either direction) would turn up in about 5 of every 100 studies like this one, so the data are only moderately surprising under no difference."

Self-test.

1. The p-value is the probability, assuming H0 and the model assumptions are true, of obtaining a test statistic at least as extreme as the one observed.
2. Type I error probability 0.05, applicable when H0 is true; Type II error probability 0.20, applicable when the specified alternative effect is true.
3. The t distribution is symmetric, so the two tails are equal and the two-sided p-value adds both. One-sided is acceptable only when the direction is fixed in advance and an effect in the other direction would be treated the same as no effect.
4. 0.05 / 12 = 0.00417. Holm controls the same FWER but is uniformly at least as powerful, because after each rejection the threshold relaxes.
5. FWER is the probability of even one false rejection; suitable for a small set of confirmatory hypotheses (for example, primary outcomes in a thesis or trial). FDR is the expected share of false rejections among all rejections; suitable for screening many outcomes (for example, many survey items or genes) where some false leads are tolerable.
6. The difference is statistically detectable but practically trivial; report the effect size and CI and say that it is below any meaningful tolerance.

Revision history