Contents / Statistics / Bayesian Inference

Chapter 10

Bayesian Inference

Prior, likelihood and posterior, and what the two frameworks actually disagree about; conjugate priors with the Beta-binomial and normal-normal updates proved, and the posterior mean shown to be a weighted average; why a flat prior is not uninformative; credible intervals against confidence intervals; the posterior predictive distribution; Bayes factors, Occam's razor and Lindley's paradox; and MCMC, with Metropolis-Hastings proved to have the posterior as its stationary distribution.

Introduction

Every chapter so far has treated the parameter as a fixed unknown constant and the data as random. That is not a mathematical necessity; it is a choice, and it is the choice that makes a confidence interval mean what it means — a statement about the long-run behaviour of a procedure, rather than about the parameter, because a fixed constant has no probability distribution to describe.

This chapter makes the other choice. Treat the parameter as a random variable with a distribution expressing what is known about it, and the whole apparatus rearranges. Bayes' theorem combines that prior distribution with the data to produce a posterior distribution, and every question is then answered by reading it off: a point estimate is a summary of the posterior, an interval is a region containing $95%$ of it, and — the thing the frequentist framework cannot deliver — the probability that the parameter lies in that interval really is $0.95$ .

Nothing here contradicts the earlier chapters. Both frameworks are internally consistent and answer different questions, and with a large sample and a prior that is not perverse they give nearly identical numbers, which the third section demonstrates. The genuine differences appear where the data are few, where information exists beyond the current sample, and where the question being asked is really about the parameter rather than about a procedure.

The cost is the prior, and it is a real cost. It must come from somewhere, two analysts may choose differently, and with small samples the choice shows in the answer. The second section is about making that choice defensibly and reporting its influence rather than hiding it. The last section covers the computation that made all of this practical: for most models the posterior has no closed form, and it was the arrival of simulation methods rather than any new mathematics that moved Bayesian inference from a philosophical position into a working toolkit.

Notation. $θ$ is the parameter, $π (θ)$ the prior, $L (θ) = f (x ∣ θ)$ the likelihood as in Statistical Inference: Estimation, and $π (θ ∣ x)$ the posterior.

10.1From Bayes' theorem to a posterior distribution

Definition 10.1 (Prior, likelihood, posterior). Given a prior density $π (θ)$ and data $x$ with likelihood $f (x ∣ θ)$ , the posterior density is

π (θ ∣ x) = \frac{f ( x ∣ θ ) π ( θ )}{f ( x )}, f (x) = \int f (x ∣ θ) π (θ) d θ,

the denominator being the marginal likelihood, a constant in $θ$ .

Because the denominator does not involve $θ$ , it serves only to make the posterior integrate to one. That licenses the notation used throughout this chapter:

π (θ ∣ x) \propto f (x ∣ θ) π (θ), posterior \propto likelihood \times prior,

and in practice one recognises the shape of the right-hand side and supplies the constant afterwards.

Intuition. The formula is a rule for updating. The prior says what was believed before; the likelihood says how well each value of $θ$ explains what was observed; the product reweights the prior in favour of the values that explain the data well.

A value of $θ$ ends up with high posterior probability only if it was plausible beforehand and predicts the data. Either factor can veto: data that a parameter value cannot produce drive its posterior to zero however plausible it seemed, and a parameter value the prior excluded stays excluded however well it would have fitted. That second point is why a prior that assigns zero probability to a region is a strong and often unwise commitment — no amount of data can ever recover from it.

Remark (What the two frameworks actually disagree about). The disagreement is not about Bayes' theorem, which is a theorem, and not about whether it applies to events — both frameworks use it freely for observable events. It is about whether a parameter may be given a probability distribution.

The frequentist position is that $θ$ is a fixed unknown constant, so $P (θ \in A)$ is either $0$ or $1$ and probability statements about it are meaningless; probability describes the sampling procedure instead. The Bayesian position is that probability describes uncertainty of any kind, including uncertainty about a constant, so a distribution over $θ$ is a legitimate description of what is known.

Each gives a coherent system. The practical consequence is what a $95%$ interval means: a confidence interval is $95%$ of the time correct across repeated samples, while a credible interval contains $95%$ of the posterior probability. The second is what most readers assume the first says, which is the most common misreading in applied statistics and is the subject of a pitfall in the testing chapter.

Example 10.2 (Updating on a single observation). A disease affects $1%$ of a population. A test has sensitivity $0.99$ and specificity $0.95$ . A randomly chosen person tests positive. Compute the posterior probability of disease.

Solution.

Prior: $P (D) = 0.01$ , so $P (D^{c}) = 0.99$ .
Likelihoods: $P (+ ∣ D) = 0.99$ and $P (+ ∣ D^{c}) = 1 - 0.95 = 0.05$ .
Marginal probability of a positive test:

P (+) = 0.99 (0.01) + 0.05 (0.99) = 0.0099 + 0.0495 = 0.0594.

Posterior: $P (D ∣ +) = 0.0099/0.0594 = 0.1667$ .
So a positive result raises the probability from $1%$ to about $17%$ — a seventeen-fold update, and still well short of a diagnosis.
Sanity check on why: the disease is rare, so the $0.99$ of healthy people generate $0.0495$ false positives, five times the $0.0099$ true positives. This is the same base-rate arithmetic as the post-test probability calculation in Statistical Inference: Hypothesis Testing, with $α$ playing the role of $1 -$ specificity and power the role of sensitivity.

□

Example 10.3 (Two tests in sequence). The same person from the previous example takes a second, independent test of the same accuracy, and it is also positive. Compute the updated probability.

Solution.

Yesterday's posterior becomes today's prior: $P (D) = 0.1667$ , so $P (D^{c}) = 0.8333$ .
Marginal probability of the second positive:

P (+_{2}) = 0.99 (0.1667) + 0.05 (0.8333) = 0.1650 + 0.0417 = 0.2067.

Posterior: $P (D ∣ +_{1}, +_{2}) = 0.1650/0.2067 = 0.7983$ .
Two positives take the probability from $1%$ to about $80%$ — the update compounds, and it compounds multiplicatively on the odds scale.
Sanity check via odds, which is faster: the prior odds are $1 : 99$ , and each positive test multiplies them by the likelihood ratio $0.99/0.05 = 19.8$ . After two tests the odds are $(1/99) (19.8)^{2} = 392.04/99 = 3.9600$ , giving a probability of $3.96/4.96 = 0.7984$ ✓.
The likelihood-ratio form shows why sequential updating is coherent: the final answer does not depend on the order the tests were processed, or on whether they were processed one at a time or together.

□

Exercises for §10.1 →

10.2Choosing a prior

The prior is the part that has no counterpart in earlier chapters, and it is where most of the objection to the approach is directed. Three considerations make the choice tractable: mathematical convenience, honest ignorance, and reporting the influence.

Definition 10.4 (Conjugate prior). A family of priors is conjugate to a likelihood if the posterior belongs to the same family. The updating is then a rule for changing the family's parameters.

Theorem 10.5 (Beta is conjugate to the binomial). If $θ \sim Beta (a, b)$ and, given $θ$ , $X \sim Binomial (n, θ)$ is observed with $x$ successes, then

θ ∣ x \sim Beta (a + x, b + n - x) .

Proof. The Beta $(a, b)$ density is proportional to $θ^{a - 1} (1 - θ)^{b - 1}$ , and the binomial likelihood is proportional to $θ^{x} (1 - θ)^{n - x}$ as a function of $θ$ . Multiplying,

π (θ ∣ x) \propto θ^{x} (1 - θ)^{n - x} \cdot θ^{a - 1} (1 - θ)^{b - 1} = θ^{(a + x) - 1} (1 - θ)^{(b + n - x) - 1},

which is the kernel of a Beta $(a + x, b + n - x)$ density. Since a density is determined by its kernel, that is the posterior.∎

The updating rule is as simple as it could be: add the successes to $a$ and the failures to $b$ . That gives the prior parameters an interpretation — $a$ and $b$ act like $a$ previously-seen successes and $b$ previously-seen failures, so $a + b$ measures how much the prior is worth in units of data. A Beta $(2, 2)$ prior carries the weight of four observations; a Beta $(50, 50)$ prior carries a hundred and will not be moved by a small experiment.

Theorem 10.6 (The posterior mean is a weighted average). Under the hypotheses above, with prior mean $μ_{0} = a / (a + b)$ and sample proportion $\overset{p}{^} = x / n$ ,

E (θ ∣ x) = \frac{a + x}{a + b + n} = w \overset{p}{^} + (1 - w) μ_{0}, w = \frac{n}{n + a + b} .

Proof. The mean of a Beta $(α, β)$ is $α / (α + β)$ , so the posterior mean is $(a + x) / (a + b + n)$ . Split the numerator, writing $x = n \overset{p}{^}$ and $a = (a + b) μ_{0}$ :

\frac{a + x}{a + b + n} = \frac{( a + b ) μ _{0} + n p ^}{n + a + b} = \frac{n}{n + a + b} \overset{p}{^} + \frac{a + b}{n + a + b} μ_{0},

and the two coefficients are $w$ and $1 - w$ , which sum to $1$ .∎

That theorem is the clearest statement of what the prior does and when it stops mattering. The posterior mean is a compromise between the prior mean and the data's own estimate, with the weight $w = n / (n + a + b)$ determined by how much data there is relative to the prior's strength. As $n \to \infty$ , $w \to 1$ and the prior washes out entirely; for $n$ small relative to $a + b$ the prior dominates. Nothing about this is hidden — the weight is computable before any data arrive.

Definition 10.7 (Uninformative and Jeffreys priors). A prior is uninformative if it is intended to let the data speak. The flat prior on $[0, 1]$ is Beta $(1, 1)$ . The Jeffreys prior is $π (θ) \propto I (θ)$ with $I$ the Fisher information of the estimation chapter; for a binomial it is Beta $(\frac{1}{2}, \frac{1}{2})$ .

Remark (Why "flat" is not the same as "uninformative"). A uniform prior looks neutral and is not, because flatness is not preserved under reparametrisation. A prior flat in $θ$ is not flat in $lo g θ$ or in the odds $θ / (1 - θ)$ — the Jacobian of the transformation reshapes it — so "know nothing" expressed on one scale becomes a definite opinion on another.

The Jeffreys prior is constructed to be invariant under reparametrisation precisely to avoid that, which is its main justification. It also frequently improper, meaning it does not integrate to a finite value; that is tolerable when the resulting posterior is proper, but it must be checked rather than assumed.

The honest practice is not to hunt for a prior with no influence but to state the prior, report how much it moved the answer, and show the result under a reasonable alternative. That is a sensitivity analysis, and it converts the prior from a hidden assumption into a reported one.

Example 10.8 (The same data under three priors). Seven successes in $10$ trials. Compute the posterior mean under a Beta $(1, 1)$ flat prior, a Beta $(2, 2)$ weakly informative prior, and a Beta $(50, 50)$ strong prior centred at $\frac{1}{2}$ .

Solution.

Beta $(1, 1)$ : posterior is Beta $(8, 4)$ , mean $8/12 = 0.6667$ . Prior weight $a + b = 2$ , so $w = 10/12 = 0.833$ .
Beta $(2, 2)$ : posterior is Beta $(9, 5)$ , mean $9/14 = 0.6429$ . Here $w = 10/14 = 0.714$ .
Beta $(50, 50)$ : posterior is Beta $(57, 53)$ , mean $57/110 = 0.5182$ . Now $w = 10/110 = 0.0909$ — the data barely register.
Check the second against the weighted-average form: $0.714 (0.7) + 0.286 (0.5) = 0.5 + 0.1429 = 0.6429$ ✓.
Sanity check on the ordering: all three lie between the prior mean $0.5$ and the MLE $0.7$ , closer to $0.7$ when the prior is weak and closer to $0.5$ when it is strong — exactly as the weight $w$ dictates. ✓
The lesson for reporting: with $n = 10$ the answer ranges from $0.52$ to $0.67$ depending on the prior, so the prior must be stated. With $n = 1000$ successes in proportion, all three would agree to two decimals.

□

Figure 10.9. Posteriors from the same data (

7

successes in

10

) under three priors of increasing strength. The flat Beta

(1, 1)

and weak Beta

(2, 2)

priors leave the posterior wide and centred near the sample proportion

0.7

; the strong Beta

(50, 50)

prior, worth a hundred prior observations, pulls the posterior back almost to

0.5

and makes it far narrower. The dashed line marks the maximum likelihood value. Prior strength is measurable in units of data, and here ten observations cannot outweigh a hundred. (The third curve is drawn as its normal approximation, indistinguishable at this scale.)

Figure 10.10. Prior, likelihood and posterior for

7

successes in

10

trials under a Beta

(2, 2)

prior. The posterior is the pointwise product of the other two, renormalised: it sits between them, closer to the likelihood because

n = 10

outweighs the prior's effective four observations. Its peak is pulled back from the maximum likelihood value of

0.7

toward the prior mean of

0.5

, and it is narrower than either input — combining two sources of information sharpens the result.

Theorem 10.11 (Normal-normal conjugacy). Let $θ \sim N (μ_{0}, τ^{2})$ and let $\overset{x}{ˉ}$ be the mean of $n$ observations from $N (θ, σ^{2})$ with $σ^{2}$ known. Then the posterior is normal with precision (reciprocal variance) equal to the sum of the prior and data precisions,

\frac{1}{τ _{n}^{2}} = \frac{1}{τ ^{2}} + \frac{n}{σ ^{2}}, μ_{n} = τ_{n}^{2} (\frac{μ _{0}}{τ ^{2}} + \frac{n x ˉ}{σ ^{2}}),

so the posterior mean is the precision-weighted average of the prior mean and the sample mean.

Proof. Drop all factors free of $θ$ . The likelihood contributes $exp {- n (\overset{x}{ˉ} - θ)^{2} / (2 σ^{2})}$ — the other part of the sum of squares does not involve $θ$ — and the prior contributes $exp {- (θ - μ_{0})^{2} / (2 τ^{2})}$ . So

π (θ ∣ x) \propto exp {- \frac{1}{2} [\frac{n ( θ - x ˉ ) ^{2}}{σ ^{2}} + \frac{( θ - μ _{0} ) ^{2}}{τ ^{2}}]} .

Expand the bracket and collect powers of $θ$ :

θ^{2} (\frac{n}{σ ^{2}} + \frac{1}{τ ^{2}}) - 2 θ (\frac{n x ˉ}{σ ^{2}} + \frac{μ _{0}}{τ ^{2}}) + const .

A quadratic $A θ^{2} - 2 Bθ$ inside $exp {- \frac{1}{2} (\cdot)}$ is the kernel of a normal with variance $1/ A$ and mean $B / A$ — complete the square to see it. Reading off $A$ and $B$ gives the stated precision and mean.∎

Intuition. Precision is the natural currency here, and the rule is as simple as addition: precisions add. The prior contributes $1/ τ^{2}$ and $n$ observations contribute $n / σ^{2}$ , and the posterior has their sum, so the posterior is always more precise than either input — information accumulates and never decreases.

The mean is then the average of the two estimates weighted by how precise each is, which is what anyone would do by hand when combining two measurements of different quality. A vague prior has $τ^{2}$ large, contributes almost no precision, and barely shifts the answer; a sharp prior contributes a lot and pulls hard.

Exercises for §10.2 →

10.3Point estimates and credible intervals

The posterior is the complete answer. Everything else in this section is a summary of it, chosen to fit what a reader can absorb.

Definition 10.12 (Posterior summaries and credible interval). The posterior mean, median and mode (the last also called the maximum a posteriori or MAP estimate) summarise the posterior's location. A $100 (1 - α) %$ credible interval is any region $C$ with

P (θ \in C ∣ x) = 1 - α,

commonly taken as the equal-tailed interval between the $α /2$ and $1 - α /2$ posterior quantiles, or as the highest posterior density region, which is the shortest such interval.

Pitfall. This is where the two frameworks differ most visibly, and the difference is worth stating in both directions.

A credible interval supports the statement most people want: given the data and the prior, there is a $95%$ probability that $θ$ lies in it. A confidence interval does not; as the estimation chapter set out, once computed it either contains $θ$ or does not, and the $95%$ describes the procedure across repeated samples.

The numbers are often nearly identical — with a flat prior and a normal model they can coincide exactly — which is why the misinterpretation so rarely causes visible damage. It is still a misinterpretation, and the honest options are to compute a credible interval and make the probability statement, or compute a confidence interval and make the procedural one.

Proposition 10.13 (The posterior mode is the MLE under a flat prior). If the prior is constant over the parameter space, the posterior is proportional to the likelihood, so the MAP estimate equals the maximum likelihood estimate.

Proof. With $π (θ) = c$ , the posterior is $π (θ ∣ x) \propto c f (x ∣ θ) \propto f (x ∣ θ)$ . Two functions proportional by a positive constant are maximised at the same point.∎

That proposition explains why the two frameworks so often agree numerically. Maximum likelihood, the workhorse of the estimation chapter, is Bayesian estimation with a flat prior and the mode as the summary — so the disagreement is about interpretation far more often than about arithmetic.

Remark (Large samples: the prior stops mattering). The Bernstein-von Mises theorem, which we quote, says that under regularity conditions the posterior converges to a normal distribution centred at the maximum likelihood estimate with variance equal to the inverse Fisher information — whatever the prior, provided it assigns positive density near the true value.

So for large $n$ the credible interval and the confidence interval converge to the same numbers, and the frequentist properties of a Bayesian procedure become good automatically. The practical reading: with abundant data the choice of framework is a matter of interpretation and convenience; with scarce data it changes the answer, and the prior must be defended.

The proviso matters. A prior assigning zero density to the true value blocks the convergence entirely, which is the formal version of the earlier warning against excluding regions.

Example 10.14 (A credible interval, and the confidence interval beside it). With the Beta $(2, 2)$ prior and $7$ successes in $10$ , the posterior is Beta $(9, 5)$ with quantiles $q_{0.025} = 0.3942$ and $q_{0.975} = 0.8567$ . Compare with the Wilson confidence interval from the estimation chapter.

Solution.

Credible interval: $(0.3942, 0.8567)$ , and the statement licensed is direct — given the prior and data, $P (0.394 < θ < 0.857 ∣ x) = 0.95$ .
Posterior mean $9/14 = 0.6429$ ; posterior mode $(9 - 1) / (9 + 5 - 2) = 8/12 = 0.6667$ , using the Beta mode formula $(α - 1) / (α + β - 2)$ .
The three summaries order themselves as mean $0.6429$ < mode $0.6667$ < MLE $0.70$ . That ordering is the general pattern: the mode tracks the likelihood most closely of the posterior summaries, while the mean is pulled furthest toward the prior, because it averages over the whole posterior including its left tail.
Wilson interval for $\overset{p}{^} = 0.7$ , $n = 10$ , $z = 1.96$ : centre $(0.7 + 0.192) /1.392 = 0.6408$ and half-width $(1.96/1.392) 0.021 + 0.0096 = 1.408 \times 0.1750 = 0.2464$ , giving $(0.394, 0.887)$ .
Sanity check on the agreement: the two intervals nearly coincide at the lower end and differ by $0.03$ at the upper. With $n = 10$ and a weak prior they are describing the same uncertainty by different routes.
What differs is what may be said. The credible interval carries a probability statement about $θ$ ; the Wilson interval carries a statement about the procedure's coverage across repeated samples.

□

Example 10.15 (Combining a prior with data on the normal scale). A quantity has prior $N (100, 25)$ . Sixteen measurements with known $σ^{2} = 100$ give $\overset{x}{ˉ} = 110$ . Find the posterior and a $95%$ credible interval.

Solution.

Precisions. Prior: $1/ τ^{2} = 1/25 = 0.04$ . Data: $n / σ^{2} = 16/100 = 0.16$ .
Posterior precision: $0.04 + 0.16 = 0.20$ , so the posterior variance is $1/0.20 = 5$ and the standard deviation is $5 = 2.2361$ .
Posterior mean:

μ_{n} = 5 (\frac{100}{25} + \frac{16 ( 110 )}{100}) = 5 (4 + 17.6) = 5 (21.6) = 108.

Credible interval: $108 \pm 1.96 (2.2361) = 108 \pm 4.3828 = (103.62, 112.38)$ .
Sanity check on the compromise: the posterior mean $108$ lies between the prior mean $100$ and the sample mean $110$ , and much closer to the sample mean because the data carry four times the precision ( $0.16$ against $0.04$ ) — the weights are $0.8$ and $0.2$ , and $0.8 (110) + 0.2 (100) = 108$ ✓.
Note the posterior standard deviation $2.24$ is smaller than the data's own standard error $σ / n = 2.5$ and smaller than the prior's $5$ . Combining information sharpens, as precision addition requires.

□

Predicting the next observation

A posterior describes the parameter. Often the question is instead about the next data point, and the two are not the same — predicting an observation must account for both uncertainty about $θ$ and the noise in the observation itself, exactly as the prediction interval of the regression chapter did.

Definition 10.16 (Posterior predictive distribution). The posterior predictive distribution of a future observation $\tilde{x}$ is

p (\tilde{x} ∣ x) = \int f (\tilde{x} ∣ θ) π (θ ∣ x) d θ,

averaging the sampling distribution over the posterior rather than fixing $θ$ at an estimate.

Theorem 10.17 (Predicting the next Bernoulli trial). Under the Beta-binomial model, the probability that the next trial is a success equals the posterior mean:

P (\tilde{X} = 1 ∣ x) = E (θ ∣ x) = \frac{a + x}{a + b + n} .

Proof. Apply the definition with $f (\tilde{x} = 1 ∣ θ) = θ$ :

P (\tilde{X} = 1 ∣ x) = \int_{0}^{1} θ π (θ ∣ x) d θ,

which is by definition the posterior mean of $θ$ . The value follows from the theorem The posterior mean is a weighted average.∎

Pitfall. Plugging a point estimate into the sampling distribution — using $f (\tilde{x} ∣ \hat{θ})$ instead of the integral — understates the uncertainty, because it treats $\hat{θ}$ as known when it is not. The error is the same one as confusing a confidence interval for a mean with a prediction interval for an observation, and it is largest exactly when the posterior is wide, which is when the data are few.

For the Bernoulli the two happen to agree on the probability of a success, as the theorem shows, but they diverge immediately for anything involving more than one future trial: the posterior predictive for the number of successes in the next $m$ trials is beta-binomial and over-dispersed relative to the binomial that a plug-in estimate would give.

Example 10.18 (Plug-in against posterior predictive). With the Beta $(9, 5)$ posterior from $7$ successes in $10$ , find the probability that the next trial succeeds, and the variance of the number of successes in the next $m = 5$ trials under both the plug-in and the predictive approach.

Solution.

Next trial. By the theorem, $P (\tilde{X} = 1 ∣ x) = 9/14 = 0.6429$ .
Plug-in for $m = 5$ . Fixing $θ = 0.6429$ gives Binomial $(5, 0.6429)$ with variance $5 (0.6429) (0.3571) = 1.1480$ .
Posterior predictive for $m = 5$ . The count is beta-binomial, whose variance is

m \overset{ˉ}{θ} (1 - \overset{ˉ}{θ}) \cdot \frac{a + b + n + m}{a + b + n + 1} = 1.1480 \times \frac{14 + 5}{15} = 1.1480 \times 1.2667 = 1.4542.

The predictive variance is $27%$ larger, because it carries the uncertainty about $θ$ in addition to the trial-to-trial noise.
Sanity check on the limit: as the posterior sharpens — larger $a + b + n$ — the inflation factor $(a + b + n + m) / (a + b + n + 1)$ tends to $1$ and the two agree. With $a + b + n = 1000$ instead of $14$ the factor would be $1.004$ .
This is the same structure as the regression chapter's prediction interval, whose extra term likewise never vanished: knowing $θ$ perfectly would still leave the coin's own randomness.

□

Exercises for §10.3 →

10.4Bayesian model comparison

Hypothesis testing has a Bayesian counterpart, and the shape of it is different enough to be worth contrasting carefully with the testing chapter.

Definition 10.19 (Bayes factor). For two models $M_{0}$ and $M_{1}$ , the Bayes factor is the ratio of marginal likelihoods

BF_{10} = \frac{P ( x ∣ M _{1} )}{P ( x ∣ M _{0} )},

and it converts prior odds into posterior odds:

posterior odds \frac{P ( M _{1} ∣ x )}{P ( M _{0} ∣ x )} = BF_{10} \times prior odds \frac{P ( M _{1} )}{P ( M _{0} )} .

Proof. Apply Bayes' theorem to each model and divide, the common marginal $P (x)$ cancelling:

\frac{P ( M _{1} ∣ x )}{P ( M _{0} ∣ x )} = \frac{P ( x ∣ M _{1} ) P ( M _{1} ) / P ( x )}{P ( x ∣ M _{0} ) P ( M _{0} ) / P ( x )} = \frac{P ( x ∣ M _{1} )}{P ( x ∣ M _{0} )} \cdot \frac{P ( M _{1} )}{P ( M _{0} )} .

∎

The Bayes factor is the factor by which the data multiply the odds, and it has three properties that a $p$ -value does not.

It can support the null: $BF_{10} = 0.1$ is ten-to-one evidence for $M_{0}$ , whereas a large $p$ -value is merely a failure to reject. It is symmetric between the two models, neither being privileged. And it automatically penalises complexity, because the marginal likelihood integrates over the parameter space — a model with more parameters spreads its prior predictive mass more thinly, so it must fit substantially better to compensate. That is an Occam's razor that falls out of the arithmetic rather than being added as a penalty term the way AIC adds one.

Remark (Lindley's paradox). For a fixed $p$ -value, the Bayes factor can favour the null more and more strongly as $n$ grows. A result significant at exactly $p = 0.05$ with $n = 1 0^{6}$ typically has a Bayes factor supporting $M_{0}$ .

The two are answering different questions rather than contradicting each other. The $p$ -value asks how surprising the data are if $M_{0}$ holds, and with a huge sample a trivial departure is surprising. The Bayes factor asks which model predicted the data better, and a tiny effect is predicted well by the null and poorly by an alternative that spread its prior over a wide range of possible effect sizes.

The practical reading: with very large samples, statistical significance and evidential support come apart, and quoting the first without an effect size is uninformative — which the testing chapter said from the frequentist side too.

Example 10.20 (A Bayes factor by hand). A coin is tested for fairness: $M_{0}$ says $θ = 0.5$ exactly, and $M_{1}$ places a uniform prior on $θ \in [0, 1]$ . In $10$ tosses, $7$ heads are observed. Compute $BF_{10}$ .

Solution.

Under $M_{0}$ : the probability of the observed count is the binomial mass at $θ = 0.5$ ,

P (x ∣ M_{0}) = (7 10) (0.5)^{10} = 120/1024 = 0.11719.

Under $M_{1}$ : integrate over the uniform prior, using the Beta integral $\int_{0}^{1} θ^{7} (1 - θ)^{3} d θ = B (8, 4) = \frac{7 ! 3 !}{11 !}$ :

P (x ∣ M_{1}) = (7 10) \int_{0}^{1} θ^{7} (1 - θ)^{3} d θ = 120 \times \frac{5040 \times 6}{39916800} = 120 \times 0.00075758 = 0.09091.

Bayes factor: $BF_{10} = 0.09091/0.11719 = 0.7758$ .
So the data mildly favour the null — the fair-coin model — by about $1.29$ to $1$ , despite $7$ heads out of $10$ looking suggestive.
Sanity check against the frequentist test: the two-sided binomial $p$ -value for $7$ or more extreme is $0.344$ , also far from significant. Both approaches say the same thing here, but only the Bayes factor states it as positive (if weak) support for fairness.
Note where $M_{1}$ 's penalty came from: it spread its prior over all of $[0, 1]$ , so it had to hedge across values like $0.1$ and $0.9$ that the data flatly contradict. A narrower alternative concentrated near $0.7$ would have produced a Bayes factor favouring $M_{1}$ .

□

Example 10.21 (Interpreting a Bayes factor against prior odds). Two models have prior odds of $1 : 4$ against $M_{1}$ (the analyst thinks it is four times less likely at the outset). The data give $BF_{10} = 12$ . Compute the posterior odds and posterior probability of $M_{1}$ .

Solution.

Prior odds: $P (M_{1}) / P (M_{0}) = 1/4 = 0.25$ .
Posterior odds $= BF_{10} \times prior odds = 12 \times 0.25 = 3$ .
Converting odds to a probability: $P (M_{1} ∣ x) = 3/ (1 + 3) = 0.75$ .
So the data have moved $M_{1}$ from a prior probability of $0.2$ to a posterior probability of $0.75$ .
Sanity check on the arithmetic of the prior: odds of $1 : 4$ correspond to $P (M_{1}) = 1/5 = 0.2$ ✓, and multiplying odds by $12$ takes $0.25$ to $3$ , which is a probability of $0.75$ ✓.
The point of separating the two factors: the Bayes factor is what the data contribute and is the same for every reader; the prior odds differ between readers, and each can combine them with their own. Reporting the Bayes factor alone is therefore the more useful thing to publish.

□

Exercises for §10.4 →

10.5Computation

Conjugate models give closed-form posteriors, and almost nothing else does. For a model with several parameters the normalising constant $\int f (x ∣ θ) π (θ) d θ$ is a high-dimensional integral with no analytic solution, and numerical quadrature fails past a few dimensions.

The resolution is to stop trying to compute the posterior and instead to sample from it. A large sample from the posterior answers every question one could ask of it: the mean is estimated by the sample mean, a credible interval by sample quantiles, and any function of $θ$ by applying it to each draw.

Definition 10.22 (Markov chain Monte Carlo). MCMC constructs a Markov chain whose stationary distribution is the posterior. Running it long enough produces draws distributed according to the posterior, from which any summary can be estimated.

Theorem 10.23 (Metropolis-Hastings has the posterior as stationary distribution). Given a current value $θ$ , propose $θ^{'}$ from a symmetric proposal density and accept with probability

α = min {1, \frac{π ( θ ^{'} ∣ x )}{π ( θ ∣ x )}},

otherwise stay at $θ$ . The resulting chain has $π (\cdot ∣ x)$ as its stationary distribution.

Proof. We verify detailed balance, which by the reversibility discussion of Markov Chains is sufficient for stationarity. For $θ \neq = θ^{'}$ the transition density is $q (θ^{'} ∣ θ) α (θ, θ^{'})$ with $q$ symmetric. Suppose without loss of generality that $π (θ^{'} ∣ x) \leq π (θ ∣ x)$ , so $α (θ, θ^{'}) = π (θ^{'} ∣ x) / π (θ ∣ x)$ and $α (θ^{'}, θ) = 1$ . Then

π (θ ∣ x) q (θ^{'} ∣ θ) α (θ, θ^{'}) = π (θ ∣ x) q (θ^{'} ∣ θ) \frac{π ( θ ^{'} ∣ x )}{π ( θ ∣ x )} = q (θ^{'} ∣ θ) π (θ^{'} ∣ x),

while the reverse flow is $π (θ^{'} ∣ x) q (θ ∣ θ^{'}) \cdot 1$ , equal to the above by symmetry of $q$ . Detailed balance holds, so $π (\cdot ∣ x)$ is stationary.∎

The decisive feature is visible in the acceptance ratio: the posterior appears only as a ratio, so the intractable normalising constant cancels. The algorithm needs only the unnormalised product likelihood $\times$ prior, which is always computable. That single observation is what made Bayesian inference practical for models of any complexity.

Remark (Gibbs sampling, and checking the chain). Gibbs sampling is the special case in which each parameter is updated in turn by drawing from its full conditional distribution given the others. When those conditionals are standard distributions — which conjugacy within each block often arranges — every proposal is accepted and no tuning is needed.

Because the draws form a Markov chain, they are not independent, and two practical consequences follow. The chain must be run past its burn-in, the initial stretch still influenced by the starting value, which is discarded. And the effective number of independent draws is smaller than the number of iterations, by a factor depending on the autocorrelation — so a chain of $10, 000$ highly correlated draws may carry the information of only a few hundred.

Convergence is assessed by running several chains from dispersed starting points and checking that they mix into the same region, formalised by the Gelman-Rubin statistic $\hat{R}$ , which compares between-chain and within-chain variance and should be near $1$ .

Example 10.24 (Monte Carlo error in a posterior summary). An MCMC run produces $10, 000$ draws from a posterior with standard deviation $2.24$ , but the autocorrelation makes the effective sample size only $400$ . Compute the Monte Carlo standard error of the posterior mean estimate, and compare with the naive figure.

Solution.

Correct. The standard error uses the effective sample size: $2.24/ 400 = 2.24/20 = 0.112$ .
Naive, using all $10, 000$ draws as if independent: $2.24/ 10000 = 2.24/100 = 0.0224$ .
The naive figure understates the Monte Carlo error by a factor of $10000/400 = 25 = 5$ .
Sanity check on whether this matters: the posterior standard deviation is $2.24$ , so even the correct Monte Carlo error of $0.112$ is only $5%$ of it — the simulation is adding little uncertainty on top of what the data leave. That is the right comparison to make, and it is the one that decides whether to run the chain longer.
Note the two error sources are different in kind. The posterior standard deviation reflects genuine uncertainty about $θ$ and does not shrink with more iterations; the Monte Carlo error reflects imperfect simulation and does.

□

Figure 10.25. Two MCMC chains from dispersed starting points. Both wander for an initial stretch — the burn-in, shaded by the dashed cut-off — before settling into the same region, which is the visual form of the convergence check that the Gelman-Rubin statistic formalises. Draws before the cut-off are discarded; draws after it are correlated neighbours, which is why the effective sample size is well below the number of iterations.

Exercises for §10.5 →

Common Mistakes

Reading a confidence interval as a credible interval. Only the second licenses "there is a $95%$ probability that $θ$ is in here"; the first is a statement about the procedure.
Believing a flat prior is uninformative. Flatness is not preserved under reparametrisation, so a prior flat in $θ$ is opinionated about $lo g θ$ .
Assigning zero prior probability to a region. No amount of data can revive it, and the Bernstein–von Mises convergence fails.
Not reporting the prior, or its influence. With small $n$ the prior moves the answer materially; report a sensitivity analysis rather than a single number.
Treating a large $p$ -value as evidence for the null. A Bayes factor can support the null; a $p$ -value can only fail to reject.
Expecting significance and evidence to agree at large $n$ . Lindley's paradox: a result at exactly $p = 0.05$ with a huge sample often favours the null on a Bayes factor.
Using all MCMC draws as if independent. The effective sample size is smaller, often by an order of magnitude, and the Monte Carlo error scales accordingly.
Failing to discard burn-in or to check convergence. Run several chains from dispersed starts and check $\hat{R}$ is near $1$ .
Assuming a Bayesian answer differs from a frequentist one. With a flat prior the MAP estimate is the MLE, and for large $n$ the intervals converge. The differences are largest exactly where the data are weakest.

About the practice questions. They check that you can carry out this chapter's computations correctly, and each one is graded on a single answer. They are not proof exercises: working through them confirms the mechanics, not that you could prove the results yourself. For that, re-read the statements above and try to reconstruct their proofs with the page closed. This chapter also has a printable problem set: proofs, sketches and find-the-error work to do on paper, with full solutions on a separate page.

Practice this chapter Printable problem set