Bayesian Inference Concept Card Template (Statistics, R and Python)

Bayesian Inference Concept Card: [Your study question or course]

A concept card for Bayesian inference: prior, likelihood and posterior, the beta-binomial and normal-normal conjugate models, credible intervals and how they differ from confidence intervals, the posterior predictive distribution, prior sensitivity, an overview of Bayes factors with cautions, and grid approximation in base R and numpy. It is for students meeting Bayesian ideas for the first time, master's and PhD students deciding whether a Bayesian analysis suits their thesis, and analysts who want to state "the probability that the rate exceeds 50%" honestly. It speeds up learning by working one model fully by formula and by code, fading the next example, and ending with self-test questions whose answers sit in a separate key. No special software is needed: everything runs in base R and numpy/scipy. The worked examples use fictional data; replace them with your own. Source basis: Bayes (1763), Jeffreys (1961), Kass and Raftery (1995), Gelman et al. Bayesian Data Analysis (named as further reading), with all text and examples written fresh.

Document control

The idea in one paragraph

Bayesian inference treats an unknown parameter as uncertain and describes that uncertainty with a probability distribution. You start with a prior distribution (what is plausible before the data), multiply it by the likelihood (how well each parameter value predicts the observed data), and rescale to get the posterior distribution (what is plausible after the data). Every answer then comes from the posterior: a best estimate, a credible interval that contains the parameter with a stated probability, the probability that the parameter exceeds a threshold, or predictions for new data. The price is that the prior must be chosen and defended, and results should be checked for sensitivity to that choice.

Intuition

Think of a weather forecaster updating a forecast. Before looking at today's radar, she has a belief based on the season (prior). The radar (data) is more or less compatible with different weather outcomes (likelihood). Her updated forecast (posterior) blends the two, leaning towards whichever source is more informative. A vague seasonal belief is quickly overruled by clear radar; a very confident belief needs a lot of radar evidence to shift. Tomorrow, today's posterior becomes the new prior: Bayesian updating is sequential learning.

The chart shows a weak Beta(2, 2) prior for the share θ of Lakemoor University (fictional) students who use the library app, and the posterior Beta(16, 8) after 14 of 20 surveyed students said yes. The data moved the centre from 0.5 to about 0.67 and made the distribution much narrower.

Formal definition

Bayes' theorem for parameters. The posterior is proportional to the prior times the likelihood; the denominator, the marginal likelihood, only rescales so the posterior integrates to 1.

$p(\theta \mid y) = \frac{p(y \mid \theta)\,p(\theta)}{p(y)}, \qquad p(y) = \int p(y \mid \theta)\,p(\theta)\,d\theta, \qquad p(\theta \mid y) \propto p(y \mid \theta)\,p(\theta)$

Beta-binomial (conjugate). With x successes in n trials and a Beta(a, b) prior on θ, the posterior is again a beta distribution: add successes to a and failures to b. The prior acts like a + b earlier observations.

$\theta \sim \text{Beta}(a, b), \quad x \mid \theta \sim \text{Binomial}(n, \theta) \;\Rightarrow\; \theta \mid x \sim \text{Beta}(a + x,\; b + n - x)$

$E[\theta \mid x] = \frac{a + x}{a + b + n} = w\,\frac{a}{a+b} + (1 - w)\,\frac{x}{n}, \qquad w = \frac{a + b}{a + b + n}$

The posterior mean is a weighted average of the prior mean and the sample proportion; the data weight grows with n.

Normal-normal (conjugate, σ known). With a normal prior on the mean μ and n observations with known SD σ, precisions (one over variances) add, and the posterior mean is a precision-weighted average.

$\mu \sim N(\mu_0, \tau_0^2), \quad y_i \mid \mu \sim N(\mu, \sigma^2) \;\Rightarrow\; \mu \mid y \sim N(\mu_n, \tau_n^2)$

$\frac{1}{\tau_n^2} = \frac{1}{\tau_0^2} + \frac{n}{\sigma^2}, \qquad \mu_n = \tau_n^2\left(\frac{\mu_0}{\tau_0^2} + \frac{n\,\bar{y}}{\sigma^2}\right)$

Credible interval. A 95% equal-tailed credible interval runs from the 2.5th to the 97.5th percentile of the posterior, so $P(L \le \theta \le U \mid y) = 0.95$. A highest-density interval (HDI) is the narrowest interval with 95% posterior probability; it differs from the equal-tailed one when the posterior is skewed.

Posterior predictive. Predictions for new data average the data model over the posterior, so they carry both sampling noise and parameter uncertainty.

$p(\tilde{y} \mid y) = \int p(\tilde{y} \mid \theta)\,p(\theta \mid y)\,d\theta$

For the beta-binomial, the number of successes in m new trials follows a beta-binomial distribution with parameters m, a + x and b + n − x.

Bayes factor. The ratio of marginal likelihoods of two models: how much more probable the data are under H1 than under H0. Posterior odds = Bayes factor × prior odds.

$BF_{10} = \frac{p(y \mid H_1)}{p(y \mid H_0)}, \qquad \frac{P(H_1 \mid y)}{P(H_0 \mid y)} = BF_{10} \times \frac{P(H_1)}{P(H_0)}$

Credible versus confidence intervals

A 95% credible interval supports the direct statement "given the model, the prior and the data, θ lies in this interval with probability 0.95". A 95% confidence interval supports only a statement about the long-run behaviour of the method (see stats-confidence-intervals). With a flat or weak prior and moderate n, the two intervals are often numerically close, but they answer different questions. When the prior is informative, they can differ substantially, and the credible interval is only as good as the prior.

Bayes factors: overview and cautions

Bayes factors compare hypotheses directly and can express evidence for H0, which a p-value cannot. Rough verbal scales exist. Jeffreys (1961) graded Bayes factors in half-powers of 10: from 1 to about 3.2 the evidence is worth only a bare mention, from 3.2 to 10 it is substantial, and above 100 it is decisive. Kass and Raftery (1995) reprinted that scale and also proposed one on the 2 log BF scale, with cut-points near BF = 3, 20 and 150 for positive, strong and very strong evidence. Cautions: Bayes factors depend strongly on the prior under H1, even when the posterior for θ barely changes; a very vague prior under H1 pushes the Bayes factor towards H0 (the Jeffreys-Lindley paradox); and verbal labels are as arbitrary as α = 0.05. Report the prior used and, where possible, a sensitivity analysis. For estimation questions, posterior intervals are usually more informative than a Bayes factor.

Worked example A: fully solved

Question. Lakemoor University (fictional) asks 20 randomly chosen students whether they use the library app; 14 say yes. What share θ of all students use it, and how probable is it that a majority do?

Step 1: likelihood. Independent yes/no answers: x = 14 successes in n = 20 binomial trials.

Step 2: prior. A weak Beta(2, 2) prior: centred on 0.5, gently ruling out values very close to 0 or 1, worth about 4 earlier observations.

Step 3: update by hand. Posterior Beta(2 + 14, 2 + 6) = Beta(16, 8). Posterior mean = 16 / 24 = 0.667. The data weight is 1 − w = 20 / 24 = 0.83, so the estimate sits 83% of the way from the prior mean 0.5 to the sample proportion 0.70.

Step 4: compute summaries. Outputs on this card are trimmed to the key lines (for example R's attr(,"conf.level") lines are dropped).

x <- 14; n <- 20; a0 <- 2; b0 <- 2
a1 <- a0 + x; b1 <- b0 + n - x
round(c(a1 = a1, b1 = b1, mean = a1 / (a1 + b1), mode = (a1 - 1) / (a1 + b1 - 2),
        sd = sqrt(a1 * b1 / ((a1 + b1)^2 * (a1 + b1 + 1))),
        lo = qbeta(0.025, a1, b1), hi = qbeta(0.975, a1, b1),
        p_gt_half = 1 - pbeta(0.5, a1, b1)), 4)
round(prop.test(14, 20, correct = FALSE)$conf.int, 4)   # frequentist Wilson CI
# Posterior predictive: how many of the next 10 students use the app?
m <- 10; k <- 0:m
pp <- choose(m, k) * beta(a1 + k, b1 + m - k) / beta(a1, b1)
round(c(mean = sum(k * pp), p_ge7 = sum(pp[k >= 7])), 4)

       a1        b1      mean      mode        sd        lo        hi p_gt_half
  16.0000    8.0000    0.6667    0.6818    0.0943    0.4708    0.8362    0.9534
[1] 0.4810 0.8545
  mean  p_ge7
6.6667 0.5601

from scipy import stats

x, n, a0, b0 = 14, 20, 2, 2
post = stats.beta(a0 + x, b0 + n - x)
print(round(post.mean(), 4), post.ppf([0.025, 0.975]).round(4), round(post.sf(0.5), 4))
pp = stats.betabinom(10, a0 + x, b0 + n - x)          # posterior predictive
print(round(pp.mean(), 4), round(pp.sf(6), 4))        # sf(6) = P(K >= 7)

0.6667 [0.4708 0.8362] 0.9534
6.6667 0.5601

Step 5: interpret. Given the weak prior and the survey, the share of app users is most plausibly around 0.67, with a 95% credible interval from 0.47 to 0.84. The posterior probability that a majority use the app is 0.95. In the next 10 students, we expect about 6.7 users, and there is a 56% chance that 7 or more of them will be users.

Step 6: compare with the frequentist view. The Wilson 95% CI is [0.48, 0.85], close to the credible interval because the prior is weak. The two-sided exact binomial test of H0: θ = 0.5 gives p = 0.115: not significant. The Bayesian summary "P(θ > 0.5 | data) = 0.95" and the p-value answer different questions, so they do not contradict each other.

Report: With a Beta(2, 2) prior, the posterior for the share of app users was Beta(16, 8): posterior mean 0.67, 95% credible interval [0.47, 0.84], P(θ > 0.5) = 0.95.

Prior sensitivity table

Always show how much the conclusions depend on the prior. Same data (14 of 20), five priors (the code continues from Example A, so x and n are already defined; the table repeats R's rounded output):

pri <- data.frame(prior = c("Beta(1,1) flat", "Beta(0.5,0.5) Jeffreys", "Beta(2,2) weak",
                            "Beta(10,10) strong at 0.5", "Beta(2,8) sceptical"),
                  a = c(1, 0.5, 2, 10, 2), b = c(1, 0.5, 2, 10, 8))
pri$post_mean <- (pri$a + x) / (pri$a + pri$b + n)
pri$lo <- qbeta(0.025, pri$a + x, pri$b + n - x)
pri$hi <- qbeta(0.975, pri$a + x, pri$b + n - x)
pri$p_gt_half <- 1 - pbeta(0.5, pri$a + x, pri$b + n - x)
cbind(pri[, 1:3], round(pri[, 4:7], 3))

The three vague priors agree closely: the data dominate. A strong prior at 0.5 (worth 20 observations, as many as the survey) pulls the mean to 0.60; a sceptical prior expecting about 20% usage pulls it to 0.53 and drops P(θ > 0.5) to 0.64. With only 20 observations, a strong prior matters, so it must be justified from real earlier evidence.

Bayes factor for the same data

Compare H0: θ = 0.5 with H1: θ ~ Beta(1, 1). Under a uniform prior the marginal probability of any count x is 1 / (n + 1).

m0 <- dbinom(14, 20, 0.5); m1 <- 1 / 21
round(c(m0 = m0, m1 = m1, BF10 = m1 / m0), 4)
m1b <- choose(20, 14) * beta(10 + 14, 10 + 6) / beta(10, 10)   # H1: Beta(10,10)
round(c(m1b = m1b, BF10b = m1b / m0), 4)

    m0     m1   BF10
0.0370 0.0476 1.2882
   m1b  BF10b
0.0593 1.6054

The Bayes factor is 1.29 with a uniform prior under H1 and 1.61 with a Beta(10, 10) prior: the data favour H1 only slightly, which is weak evidence on either scale above, and the value visibly depends on the H1 prior. The posterior probability of a majority (0.95) and the Bayes factor (about 1.3) answer different questions: the first asks where θ is, the second whether θ is exactly 0.5.

Worked example B: half solved

Question. Brightwell Bakery (fictional) weighs 12 sourdough loaves (mean 497.55 g). The scale's and process's SD is known from long experience to be σ = 2.5 g. Before today, the head baker believed the batch mean was near 500 g, give or take about 4 g (two prior SDs): prior μ ~ N(500, 2²). What is the posterior for the batch mean?

sigma <- 2.5; mu0 <- 500; tau0 <- 2
w <- c(497.2, 501.5, 494.8, 498.9, 496.1, 499.4, 495.7, 500.2, 497.8, 493.9, 498.5, 496.6)
n <- length(w); ybar <- mean(w)
prec <- 1 / tau0^2 + n / sigma^2
mun <- (mu0 / tau0^2 + n * ybar / sigma^2) / prec; taun <- sqrt(1 / prec)
round(c(prior_prec = 1 / tau0^2, data_prec = n / sigma^2, post_prec = prec,
        post_mean = mun, post_sd = taun, lo = mun - 1.96 * taun, hi = mun + 1.96 * taun,
        p_below_500 = pnorm(500, mun, taun), weight_data = (n / sigma^2) / prec), 4)

 prior_prec   data_prec   post_prec   post_mean     post_sd          lo
     0.2500      1.9200      2.1700    497.8323      0.6788    496.5017
         hi p_below_500 weight_data
   499.1628      0.9993      0.8848

import numpy as np
from scipy import stats

sigma, mu0, tau0 = 2.5, 500, 2
w = np.array([497.2, 501.5, 494.8, 498.9, 496.1, 499.4, 495.7, 500.2, 497.8, 493.9, 498.5, 496.6])
prec = 1 / tau0**2 + w.size / sigma**2
mun = (mu0 / tau0**2 + w.size * w.mean() / sigma**2) / prec; taun = prec**-0.5
print(round(mun, 4), round(taun, 4), stats.norm.ppf([0.025, 0.975], mun, taun).round(4),
      round(stats.norm.cdf(500, mun, taun), 4))

497.8323 0.6788 [496.5017 499.1628] 0.9993

1. Prior precision = 1 / 2² = [ ]; data precision = 12 / 2.5² = [ ].
2. Posterior precision = [ ]; posterior SD = [ ] g.
3. Weight on the data = data precision / posterior precision = [ ]; so the posterior mean sits [ ]% of the way from 500 to 497.55.
4. Posterior mean = [ ] g; 95% credible interval [ ] to [ ] g.
5. P(μ < 500 | data) = [ ]. Write one sentence for the bakery manager using this number.
6. Compare with the frequentist t-interval [496.11, 498.99] (stats- hypothesis-testing, Example A): why is the credible interval [narrower / wider], and why is its centre [closer to / further from] 500? [ ].

Exercise C: unsolved

Harbour City Library (fictional) checks 15 returned books and finds 3 returned late. Use a flat Beta(1, 1) prior for the late-return rate θ.

1. Write down the posterior distribution.
2. Compute the posterior mean.
3. Find the 95% equal-tailed credible interval (use qbeta or stats.beta.ppf).
4. Compute P(θ < 0.3 | data).
5. A colleague says, "So there is a 95% chance θ is in that interval." Is that correct for this interval? Would it be correct for a confidence interval?
6. How would a Beta(5, 20) prior from last year's audit change the posterior mean? Compute it.

Simulation you can run

Grid approximation. For one parameter, chop its range into many points, compute prior × likelihood at each, and normalise. No conjugacy or special software needed. It reproduces the exact Beta(16, 8) answer and then handles a non-conjugate triangular prior just as easily.

grid <- seq(0, 1, length.out = 1001)
prior <- dbeta(grid, 2, 2)
lik <- dbinom(14, 20, grid)
post <- prior * lik; post <- post / sum(post)
cdf <- cumsum(post)
round(c(mean = sum(grid * post), lo = grid[which(cdf >= 0.025)[1]],
        hi = grid[which(cdf >= 0.975)[1]], p_gt_half = sum(post[grid > 0.5])), 4)

prior2 <- 1 - abs(grid - 0.5) * 2                  # triangular prior peaked at 0.5
post2 <- prior2 * lik; post2 <- post2 / sum(post2); cdf2 <- cumsum(post2)
round(c(mean = sum(grid * post2), lo = grid[which(cdf2 >= 0.025)[1]],
        hi = grid[which(cdf2 >= 0.975)[1]]), 4)

set.seed(2026)                                     # Monte Carlo from the posterior
th <- rbeta(10000, 16, 8)
round(c(mean = mean(th), p_gt_half = mean(th > 0.5), quantile(th, c(0.025, 0.975))), 4)

     mean        lo        hi p_gt_half
   0.6667    0.4710    0.8360    0.9530
  mean     lo     hi
0.6546 0.4650 0.8280
     mean p_gt_half      2.5%     97.5%
   0.6660    0.9499    0.4665    0.8351

import numpy as np
from scipy import stats

grid = np.linspace(0, 1, 1001)
prior = stats.beta.pdf(grid, 2, 2)
lik = stats.binom.pmf(14, 20, grid)
post = prior * lik; post /= post.sum(); cdf = np.cumsum(post)
print(round((grid * post).sum(), 4), round(grid[np.argmax(cdf >= 0.025)], 3),
      round(grid[np.argmax(cdf >= 0.975)], 3), round(post[grid > 0.5].sum(), 4))

rng = np.random.default_rng(2026)
th = rng.beta(16, 8, 10000)
print(round(th.mean(), 4), round((th > 0.5).mean(), 4), np.quantile(th, [0.025, 0.975]).round(4))

0.6667 0.471 0.836 0.953
0.667 0.9532 [0.4712 0.8351]

The grid answer (mean 0.6667, interval [0.471, 0.836], P(θ > 0.5) = 0.953) matches the exact beta results to the grid's resolution of 0.001. The triangular prior gives a slightly lower mean (0.655) because it puts more weight near 0.5. Sampling 10,000 draws from the posterior gives P(θ > 0.5) of 0.950 in R and 0.953 in Python: Monte Carlo answers wobble around the exact 0.9534, and R and Python generators differ even with the same seed. Posterior sampling is the idea behind MCMC.

Next steps. Grids become impractical beyond two or three parameters, because the number of grid points explodes. For regression and hierarchical models, use Markov chain Monte Carlo through Stan (via rstan, cmdstanr or brms in R, cmdstanpy in Python) or PyMC in Python, and check convergence diagnostics such as R-hat and effective sample size.

Misconceptions

Connections to other concepts

• Probability basics (stats-probability-basics): Bayes' theorem for events is the same rule applied to parameters.
• Maximum likelihood (stats-estimation-mle): the likelihood is shared; with a flat prior, the posterior mode equals the MLE.
• Confidence intervals (stats-confidence-intervals): credible intervals answer the question many people think a confidence interval answers.
• Hypothesis testing (stats-hypothesis-testing): Bayes factors and posterior probabilities are alternatives to p-values, with their own cautions.
• Distributions (stats-distributions-sheet): beta, binomial, normal and beta-binomial are the building blocks here.
• Analysis plans (stats-analysis-plan, science/preregistration): priors, like α and hypotheses, belong in the plan before the data.

Self-test

1. State Bayes' theorem for a parameter θ and name each part.
2. A Beta(3, 7) prior meets 9 successes in 10 trials. What is the posterior, and what is its mean?
3. Explain in one sentence why the posterior mean in the beta-binomial model is a compromise between the prior mean and x / n.
4. In the normal-normal model, what happens to the posterior when n grows very large? When the prior SD τ0 grows very large?
5. Give one sentence that is true of a 95% credible interval but not of a 95% confidence interval.
6. Name two cautions when reporting a Bayes factor.

My notes and next steps

Sources and further reading

Structure aligned with a first Bayesian course (prior, likelihood, posterior, prediction, model checking) and the GAISE College Report (ASA 2016) emphasis on simulation; based on the works below. All wording and examples are original.

• Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53, 370–418.
• Jeffreys, H. (1961). Theory of Probability (3rd ed.). Clarendon Press.
• Kass, R. E., and Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795.
• Lindley, D. V. (1957). A statistical paradox. Biometrika, 44(1–2), 187–192.
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). CRC Press.
• McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and Stan (2nd ed.). CRC Press.
• Kruschke, J. K. (2015). Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan (2nd ed.). Academic Press.
• GAISE College Report ASA Revision Committee (2016). Guidelines for Assessment and Instruction in Statistics Education: College Report 2016. American Statistical Association.
• Software next steps: Stan (mc-stan.org), brms, PyMC.
• Further reading (by name only): de Smith, M. J. (2024). Statistical Analysis Handbook (2024 edition, StatsRef); OpenIntro Statistics (CC BY-SA).

Answer key

Example B.

1. Prior precision 0.25; data precision 1.92.
2. Posterior precision 2.17; posterior SD = 1 / √2.17 = 0.679 g.
3. Weight on data 0.885, so the posterior mean sits about 88% of the way from 500 to 497.55.
4. Posterior mean 497.83 g; 95% credible interval [496.50, 499.16] g.
5. 0.9993. Example: "Given what we believed before and today's weighings, it is almost certain (over 99.9%) that this batch averages below the 500 g label."
6. Narrower, because σ is treated as known and the prior adds information; the centre is closer to 500 because the prior, centred on 500, pulls the estimate towards it (shrinkage).

Exercise C.

1. Beta(1 + 3, 1 + 12) = Beta(4, 13).
2. 4 / 17 = 0.235.
3. [0.073, 0.457] (R qbeta(c(0.025, 0.975), 4, 13)).
4. pbeta(0.3, 4, 13) = 0.754.
5. Yes for the credible interval (given the flat prior and the model); no for a confidence interval, whose 95% refers to the procedure.
6. Beta(5 + 3, 20 + 12) = Beta(8, 32); posterior mean 8 / 40 = 0.200, pulled towards last year's rate of 0.2.

Self-test.

1. $p(\theta \mid y) = p(y \mid \theta)\,p(\theta) / p(y)$: posterior, likelihood, prior, marginal likelihood.
2. Beta(12, 8); mean 12 / 20 = 0.60.
3. It equals a weighted average of a / (a + b) and x / n with weights proportional to a + b and n.
4. Large n: the data precision dominates and the posterior concentrates around $\bar{y}$ with SD near σ / √n. Large τ0: the prior precision tends to 0, so the posterior approaches N($\bar{y}$, σ²/n).
5. "Given the prior and model, the parameter lies in this interval with probability 0.95."
6. It depends strongly on the prior under H1 (very vague priors favour H0), and it gives posterior odds only after multiplying by prior odds; verbal labels are arbitrary.

Revision history