Contents / Statistics / Logistic Regression

Chapter 8

Logistic Regression

Why a linear model fails for a binary response; the logistic function and the logit link; coefficients as log odds ratios and why they are not probability changes; maximum likelihood with a concave log-likelihood, the score equations and iteratively reweighted least squares; deviance and likelihood ratio tests; separation, sample size and linearity in the logit; assessing a classifier by confusion matrix, ROC and calibration; and generalised linear models including Poisson, multinomial and ordinal regression.

Introduction

The previous chapter modelled a quantitative response. A great many responses are not quantitative: a patient recovers or does not, a customer buys or does not, an email is spam or is not. The natural target is then a probability, and probabilities are awkward in a way that forces a change of model rather than a patch on the old one.

The awkwardness is that probabilities live in $[0, 1]$ and a linear function does not. Fit a straight line to a binary response and it will predict negative probabilities at one end and probabilities above one at the other, not as a rare pathology but as a certainty for extreme enough predictors. Two further problems come with it, and the first section sets out all three before proposing anything, because the shape of the fix is determined by the shape of the failure.

The fix is to model a transformed version of the probability — the log-odds — linearly, which maps $[0, 1]$ onto the whole real line and so removes the range problem at a stroke. What that buys, and what it costs in interpretability, is the business of the second section: coefficients become log odds ratios, which is less immediate than "units of $y$ per unit of $x$ " and is exactly as precise.

Estimation changes too. Least squares no longer applies, and there is no closed-form solution at all; the coefficients are found by maximum likelihood, iteratively. That sounds like a loss and is largely a gain, because the likelihood machinery of the estimation chapter comes with it — the log-likelihood turns out to be concave, so the iteration cannot get stuck, and likelihood-ratio tests replace the $F$ -tests of the linear model.

The final section places logistic regression in the larger family it belongs to. It is one generalised linear model among several, differing from ordinary regression only in two choices — a distribution for the response and a link function — and once that is seen, Poisson regression for counts and the ordinal and multinomial extensions require no new theory.

Notation follows the regression chapter: $X$ is the design matrix, $β$ the coefficient vector, and $x_{i}^{⊤}$ the $i$ -th row. The response $Y_{i} \in {0, 1}$ , and $p_{i} = P (Y_{i} = 1 ∣ x_{i})$ . Maximum likelihood facts are from Statistical Inference: Estimation.

8.1Why a straight line will not do

Suppose we fit $Y_{i} = β_{0} + β_{1} x_{i} + ε_{i}$ by least squares with $Y_{i}$ binary. This is the linear probability model, and since $E (Y_{i}) = p_{i}$ for a binary variable, it is a model for the probability. Three things go wrong, of increasing seriousness.

Proposition 8.1 (Three failures of the linear probability model). For a binary response modelled as $E (Y_{i}) = β_{0} + β_{1} x_{i}$ with $β_{1} \neq = 0$ :

the fitted probability leaves $[0, 1]$ for all sufficiently large or small $x$ ;
the error variance is $p_{i} (1 - p_{i})$ , which depends on $x_{i}$ , so the errors are heteroscedastic;
the errors are not normal — for given $x_{i}$ the error takes only the two values $1 - p_{i}$ and $- p_{i}$ .

Proof.

The fitted value $β_{0} + β_{1} x$ is an unbounded linear function of $x$ ; with $β_{1} \neq = 0$ it exceeds $1$ once $x > (1 - β_{0}) / β_{1}$ and falls below $0$ once $x < - β_{0} / β_{1}$ (with the inequalities reversed for $β_{1} < 0$ ).
$Y_{i}$ is Bernoulli $(p_{i})$ , so $Var (Y_{i}) = p_{i} (1 - p_{i})$ , and $ε_{i} = Y_{i} - p_{i}$ inherits it. Since $p_{i}$ varies with $x_{i}$ , so does the variance.
$ε_{i} = Y_{i} - p_{i}$ takes the value $1 - p_{i}$ with probability $p_{i}$ and $- p_{i}$ with probability $1 - p_{i}$ . A two-point distribution is not normal.

∎

Example 8.2 (Watching the linear probability model fail). A linear probability model of purchase on income (in thousands) is fitted as $\overset{p}{^} = - 0.15 + 0.008 x$ . Find the income range over which it produces valid probabilities, and evaluate it at $$20, 000$ and $$160, 000$ .

Solution.

Valid range. $\overset{p}{^} \geq 0$ requires $x \geq 0.15/0.008 = 18.75$ ; $\overset{p}{^} \leq 1$ requires $x \leq 1.15/0.008 = 143.75$ . So the model is only usable for incomes between $$18, 750$ and $$143, 750$ .
At $x = 20$ : $\overset{p}{^} = - 0.15 + 0.16 = 0.01$ , just inside the range and plausible.
At $x = 160$ : $\overset{p}{^} = - 0.15 + 1.28 = 1.13$ — a probability of $113%$ .
The problem is not that the estimate is imprecise; no confidence interval around $1.13$ can help, because the quantity being estimated cannot exceed $1$ . This is failure (1) of the proposition, and it is why a new functional form is needed rather than a correction.
Sanity check on the variance failure too: at $x = 20$ the fitted $p = 0.01$ gives error variance $0.01 (0.99) = 0.0099$ , while at $x = 80$ the fitted $p = 0.49$ gives $0.2499$ — a twenty-five-fold difference across the data, so a single $σ^{2}$ describes neither.
A logistic fit to the same data cannot produce either pathology: its predictions approach $0$ and $1$ asymptotically and never cross them.

□

The three failures are not equally damaging, and ranking them is what determines the remedy. Heteroscedasticity and non-normality are familiar complaints from the regression chapter: by Gauss-Markov the estimates stay unbiased, and only the standard errors are wrong, which weighted least squares or robust standard errors could repair.

The first failure is different in kind. A model that predicts a probability of $1.3$ is not imprecise; it is making a statement that cannot be true, and no adjustment of standard errors helps. That is the failure that forces a new model, and the fix is to choose a functional form that cannot leave $[0, 1]$ in the first place.

Definition 8.3 (The logistic function and the logit). The logistic (sigmoid) function and its inverse, the logit, are

σ (z) = \frac{1}{1 + e ^{- z}} = \frac{e ^{z}}{1 + e ^{z}}, logit (p) = lo g \frac{p}{1 - p} .

Proposition 8.4 (Properties of the logistic function). $σ$ maps $R$ onto $(0, 1)$ , is strictly increasing, satisfies $σ (0) = \frac{1}{2}$ and $σ (- z) = 1 - σ (z)$ , and has derivative

σ^{'} (z) = σ (z) (1 - σ (z)) .

It is the inverse of the logit: $logit (σ (z)) = z$ .

Proof. Since $e^{- z} > 0$ always, the denominator $1 + e^{- z}$ exceeds $1$ , so $σ (z) \in (0, 1)$ ; as $z \to \infty$ the denominator tends to $1$ and as $z \to - \infty$ it grows without bound, giving the limits $1$ and $0$ .

For the derivative, differentiate $σ (z) = (1 + e^{- z})^{- 1}$ by the chain rule:

σ^{'} (z) = - (1 + e^{- z})^{- 2} \cdot (- e^{- z}) = \frac{e ^{- z}}{( 1 + e ^{- z} ) ^{2}} = \frac{1}{1 + e ^{- z}} \cdot \frac{e ^{- z}}{1 + e ^{- z}} = σ (z) (1 - σ (z)),

the last step because $\frac{e ^{- z}}{1 + e ^{- z}} = 1 - \frac{1}{1 + e ^{- z}}$ . Positivity of $σ^{'}$ gives strict monotonicity.

For the inverse, with $p = σ (z)$ we have $1 - p = e^{- z} / (1 + e^{- z})$ , so $p / (1 - p) = e^{z}$ and $lo g {p / (1 - p)} = z$ .∎

The identity $σ^{'} = σ (1 - σ)$ is not a curiosity — it is the Bernoulli variance $p (1 - p)$ , and it will reappear as the weight in the estimation section. That the derivative of the mean function equals the variance function is characteristic of the exponential family and is what makes these models tractable.

Figure 8.5. The logistic curve against a fitted straight line for the same binary data (points at

y = 0

and

y = 1

). The line leaves the unit interval at both ends, predicting probabilities below

0

and above

1

; the logistic curve approaches the bounds without reaching them. Both are steepest in the middle, but the logistic flattens where the outcome is nearly certain, which is the behaviour a probability should have — moving a predictor further does little once the answer is already almost settled.

Exercises for §8.1 →

8.2The model, and what its coefficients mean

Definition 8.6 (The logistic regression model). With $p_{i} = P (Y_{i} = 1 ∣ x_{i})$ , the model is linear on the log-odds scale:

logit (p_{i}) = lo g \frac{p _{i}}{1 - p _{i}} = x_{i}^{⊤} β, equivalently p_{i} = σ (x_{i}^{⊤} β),

with the $Y_{i}$ independent Bernoulli given the predictors.

The linearity is on the logit scale, not on the probability scale, and everything that is initially confusing about logistic regression follows from that. A one-unit change in $x_{j}$ changes the log-odds by exactly $β_{j}$ regardless of where you start; it changes the probability by an amount that depends on where you start, being largest near $p = \frac{1}{2}$ and vanishing near $0$ and $1$ .

Definition 8.7 (Odds and odds ratio). The odds of an event of probability $p$ are $p / (1 - p)$ . For two groups with probabilities $p_{1}, p_{2}$ , the odds ratio is

OR = \frac{p _{1} / ( 1 - p _{1} )}{p _{2} / ( 1 - p _{2} )} .

Theorem 8.8 (Coefficients are log odds ratios). In the logistic model, increasing $x_{j}$ by one unit with all other predictors held fixed multiplies the odds by $e^{β_{j}}$ . That is, $e^{β_{j}}$ is the odds ratio associated with a one-unit increase in $x_{j}$ , and $β_{j} = lo g (OR)$ .

Proof. Write $η = x^{⊤} β$ for the linear predictor. By the definition of the logit, the odds are

\frac{p}{1 - p} = e^{η} .

Increasing $x_{j}$ by one with the others fixed replaces $η$ by $η + β_{j}$ , so the odds become $e^{η + β_{j}} = e^{η} e^{β_{j}}$ . The ratio of the new odds to the old is therefore $e^{β_{j}}$ , independent of the starting value of $η$ .∎

The independence of the starting point is the whole reason the logit scale was chosen. On the probability scale no such constant statement exists — which is why "increases the probability by $β_{j}$ " is never a correct reading of a logistic coefficient.

Pitfall. An odds ratio is not a relative risk, and the two diverge badly for common outcomes. If a treatment moves a probability from $0.4$ to $0.6$ , the relative risk is $0.6/0.4 = 1.5$ while the odds ratio is $(0.6/0.4) / (0.4/0.6) = 2.25$ . Reporting the odds ratio in the language of risk overstates the effect by half again.

The two agree only when the outcome is rare, since then $1 - p \approx 1$ and the odds $p / (1 - p) \approx p$ . For an outcome with probability $0.01$ the two differ by about $1%$ ; for one with probability $0.5$ they differ enormously.

Example 8.9 (Reading a fitted logistic model). A model of loan default gives $logit (\overset{p}{^}) = - 4 + 0.05 (age of debt in months)$ . Compute the probability of default at $60$ months, the odds ratio per month, and per year.

Solution.

Linear predictor at $x = 60$ : $η = - 4 + 0.05 (60) = - 4 + 3 = - 1$ .
Probability: $\overset{p}{^} = σ (- 1) = 1/ (1 + e^{1}) = 1/3.71828 = 0.2689$ .
Odds ratio per month: $e^{0.05} = 1.0513$ , so each extra month multiplies the odds of default by about $1.05$ — a $5.1%$ increase in the odds.
Per year: twelve months multiply the log-odds by $12 \times 0.05 = 0.6$ , so the odds ratio is $e^{0.6} = 1.8221$ — the odds nearly double over a year.
Sanity check on the non-linearity: at $x = 80$ , $η = 0$ and $\overset{p}{^} = 0.5$ ; at $x = 100$ , $η = 1$ and $\overset{p}{^} = 0.7311$ . The same $20$ -month increase raises the probability by $0.231$ in the first step and by $0.231$ in the second — symmetric about $p = \frac{1}{2}$ , as $σ (- z) = 1 - σ (z)$ requires ✓ — but a further $20$ months to $x = 120$ gives $\overset{p}{^} = 0.8808$ , a rise of only $0.150$ . The probability effect shrinks as the curve flattens while the odds effect stays exactly constant.

□

Intuition. Why odds rather than probabilities? Because odds multiply cleanly and probabilities do not.

Probabilities are bounded above, so "doubling" one is not always possible — you cannot double $0.7$ . Odds range over $(0, \infty)$ with no upper bound, so multiplying them is always meaningful, and taking logarithms turns that multiplication into addition. A model that is additive in log-odds is therefore multiplicative in odds, and can never produce an impossible value however extreme the predictors.

The price is that the natural summary is a ratio of odds, which most people do not have an intuition for. That is a real cost, and it is why reporting predicted probabilities at a few representative predictor values is usually more informative to a reader than the coefficient table.

Figure 8.10. Equal steps in the linear predictor produce very unequal steps in probability. Each marked point is one unit further along

η

, and the vertical gaps between consecutive points shrink as the curve flattens: from

η = 0

1

the probability rises by

0.231

, from

2

3

by only

0.072

, and from

4

5

0.011

. The odds ratio is the same

e^{1} = 2.72

at every step. This is why a logistic coefficient has no fixed meaning on the probability scale.

Example 8.11 (The same coefficient, three different probability effects). For a model with $β_{1} = 0.8$ , compute the change in probability produced by a one-unit increase in $x$ starting from a linear predictor of $η = - 2$ , $η = 0$ and $η = 2$ .

Solution.

From $η = - 2$ : $σ (- 2) = 0.1192$ and $σ (- 1.2) = 0.2315$ , a rise of $0.1123$ .
From $η = 0$ : $σ (0) = 0.5$ and $σ (0.8) = 0.6900$ , a rise of $0.1900$ .
From $η = 2$ : $σ (2) = 0.8808$ and $σ (2.8) = 0.9427$ , a rise of $0.0619$ .
The same coefficient produces probability changes of $0.112$ , $0.190$ and $0.062$ — differing by a factor of three depending only on the starting point.
What is constant: the odds ratio $e^{0.8} = 2.2255$ at all three. Check the middle case directly: odds go from $0.5/0.5 = 1$ to $0.69/0.31 = 2.2258$ ✓.
Sanity check on the pattern: the largest effect is at $η = 0$ , where the logistic curve is steepest ( $σ^{'} = 0.25$ , its maximum), and it falls away symmetrically. That is the derivative $σ (1 - σ)$ of the proposition Properties of the logistic function, visible as a rate of change.

□

Exercises for §8.2 →

8.3Fitting by maximum likelihood

Least squares does not apply: there is no constant-variance normal error to minimise squares of. Maximum likelihood does, and the model was in fact constructed so that it would.

Theorem 8.12 (The log-likelihood and the score equations). With $p_{i} = σ (x_{i}^{⊤} β)$ , the log-likelihood is

ℓ (β) = i = 1 \sum n [y_{i} x_{i}^{⊤} β - lo g (1 + e^{x_{i}^{⊤} β})],

and its gradient is

\nabla ℓ (β) = i = 1 \sum n (y_{i} - p_{i}) x_{i} = X^{⊤} (y - p) .

Proof. Each $Y_{i}$ is Bernoulli $(p_{i})$ , so its contribution to the likelihood is $p_{i}^{y_{i}} (1 - p_{i})^{1 - y_{i}}$ and to the log-likelihood

y_{i} lo g p_{i} + (1 - y_{i}) lo g (1 - p_{i}) = y_{i} lo g \frac{p _{i}}{1 - p _{i}} + lo g (1 - p_{i}) .

The first term is $y_{i} x_{i}^{⊤} β$ by the definition of the model. For the second, $1 - p_{i} = 1 - σ (η_{i}) = 1/ (1 + e^{η_{i}})$ , so $lo g (1 - p_{i}) = - lo g (1 + e^{η_{i}})$ . Summing gives $ℓ$ .

For the gradient, differentiate with respect to $β$ using $\partial η_{i} / \partial β = x_{i}$ :

\nabla ℓ = i \sum [y_{i} x_{i} - \frac{e ^{η_{i}}}{1 + e ^{η_{i}}} x_{i}] = i \sum (y_{i} - p_{i}) x_{i},

since $e^{η} / (1 + e^{η}) = σ (η) = p_{i}$ .∎

The score equations $X^{⊤} (y - p) = 0$ are worth comparing with the normal equations $X^{⊤} (y - \hat{y}) = 0$ of the linear model. They have exactly the same form: the residuals are orthogonal to the predictors. The difference is that $p$ depends on $β$ non-linearly, so the equations cannot be solved in closed form.

Theorem 8.13 (The log-likelihood is concave). The Hessian of the log-likelihood is

\nabla^{2} ℓ (β) = - X^{⊤} WX, W = diag (p_{i} (1 - p_{i})),

which is negative semidefinite. Hence $ℓ$ is concave, and if $X$ has full rank and the $p_{i}$ lie strictly in $(0, 1)$ it is strictly concave, so any stationary point is the unique global maximum.

Proof. Differentiate the gradient. Since $\partial p_{i} / \partial β = σ^{'} (η_{i}) x_{i} = p_{i} (1 - p_{i}) x_{i}$ by the proposition Properties of the logistic function,

\nabla^{2} ℓ = - i \sum p_{i} (1 - p_{i}) x_{i} x_{i}^{⊤} = - X^{⊤} WX .

For any vector $v$ , writing $u = Xv$ ,

v^{⊤} (X^{⊤} WX) v = u^{⊤} Wu = i \sum p_{i} (1 - p_{i}) u_{i}^{2} \geq 0,

every weight being non-negative. So $- \nabla^{2} ℓ$ is positive semidefinite and $ℓ$ is concave. Under full rank and $0 < p_{i} < 1$ the weights are strictly positive and $u = 0$ forces $v = 0$ , giving strict concavity.∎

Concavity is the guarantee that makes the iterative fit trustworthy: there are no local maxima to get trapped in, and Newton's method converges from essentially any starting point. The standard algorithm, iteratively reweighted least squares, is Newton's method written suggestively — each step solves a weighted least-squares problem with weights $p_{i} (1 - p_{i})$ and an adjusted response, so a logistic fit is a sequence of linear fits.

Example 8.14 (One Newton step by hand). A single-parameter logistic model $logit (p) = β$ (intercept only) is fitted to $n = 10$ observations of which $7$ are successes. Starting from $β^{(0)} = 0$ , take one Newton step and compare with the exact maximiser.

Solution.

At $β^{(0)} = 0$ , every $p_{i} = σ (0) = 0.5$ .
The gradient is $\sum_{i} (y_{i} - p_{i}) = 7 - 10 (0.5) = 2$ .
The Hessian is $- \sum_{i} p_{i} (1 - p_{i}) = - 10 (0.25) = - 2.5$ .
Newton's step is $β^{(1)} = β^{(0)} - (\nabla^{2} ℓ)^{- 1} \nabla ℓ = 0 - (- 2.5)^{- 1} (2) = 0.8$ .
Exact answer. For an intercept-only model the MLE of $p$ is the sample proportion $0.7$ — the Bernoulli MLE from the estimation chapter, transported through invariance — so $\hat{β} = logit (0.7) = lo g (7/3) = 0.8473$ .
Sanity check: one step moved from $0$ to $0.8$ , most of the way to $0.8473$ . A second step would land within about $0.001$ ; Newton's method converges quadratically near the optimum, and by the theorem The log-likelihood is concave there is nowhere else for it to go.

□

Definition 8.15 (Deviance). The deviance of a fitted model is

D = - 2 ℓ (\hat{β}),

for binary data, measuring lack of fit; smaller is better. It plays the role $SSE$ plays in the linear model.

Theorem 8.16 (Likelihood ratio test for nested models). For a reduced model nested in a full model differing by $k$ parameters, under the null that the extra coefficients are all zero,

D_{red} - D_{full} = 2 [ℓ_{full} - ℓ_{red}] d χ_{k}^{2},

rejecting for large values.

Proof. This is Wilks' theorem, which we quote: twice the log-likelihood ratio between nested models converges in distribution to $χ^{2}$ with degrees of freedom equal to the difference in the number of free parameters. The equality of the two expressions is immediate from the definition of deviance, the factors of $- 2$ cancelling.∎

This is the exact analogue of the nested $F$ -test of the regression chapter, with deviance in place of $SSE$ and a $χ^{2}$ reference in place of $F$ . Individual coefficients can also be tested by the Wald statistic $\hat{β}_{j} / SE (\hat{β}_{j})$ , referred to a standard normal, with standard errors from the diagonal of $(X^{⊤} WX)^{- 1}$ — the observed information, exactly as the Cramér–Rao discussion of the estimation chapter would predict.

Example 8.17 (Testing a fitted model). A model of default with two predictors has deviance $412.6$ ; dropping both gives deviance $431.9$ . Test at $α = 0.05$ using $χ_{0.05, 2}^{2} = 5.991$ . Separately, one coefficient is $0.05$ with standard error $0.018$ — test it.

Solution.

Likelihood ratio test. The difference is $431.9 - 412.6 = 19.3$ on $k = 2$ degrees of freedom.
Since $19.3 > 5.991$ , reject: the two predictors jointly carry information about default.
Wald test. $z = 0.05/0.018 = 2.778$ , and $∣2.778∣ > 1.96$ , so that coefficient is significant at the $5%$ level.
Confidence interval for the coefficient: $0.05 \pm 1.96 (0.018) = 0.05 \pm 0.0353 = (0.0147, 0.0853)$ .
On the odds-ratio scale, exponentiate the endpoints — legitimate because $exp$ is increasing, so the coverage is unchanged: $(e^{0.0147}, e^{0.0853}) = (1.0148, 1.0890)$ .
Sanity check on the interval: it excludes $1$ , agreeing with the significant Wald test ✓. Note that the interval is computed on the log-odds scale and then transformed, never the reverse — the sampling distribution is approximately normal for $\hat{β}$ , not for $e^{\hat{β}}$ , which is skewed.

□

Exercises for §8.3 →

8.4Assumptions, and the failure modes peculiar to this model

Definition 8.18 (Assumptions of logistic regression).

Independence of the observations.
Linearity in the logit — the log-odds is linear in the predictors.
No perfect multicollinearity, so $X$ has full rank.
Adequate sample size relative to the number of predictors and the rarity of the outcome.

Note what is absent: there is no normality assumption and no equal-variance assumption, because the response distribution is Bernoulli by construction and its variance $p_{i} (1 - p_{i})$ is determined by the mean rather than being a free parameter. Two of the four LINE conditions simply do not arise.

Theorem 8.19 (Separation destroys the maximum likelihood estimate). If there is a vector $b$ such that $x_{i}^{⊤} b > 0$ whenever $y_{i} = 1$ and $x_{i}^{⊤} b < 0$ whenever $y_{i} = 0$ — the data are completely separated — then the likelihood has no maximum: $ℓ (t b) \to 0$ from below as $t \to \infty$ , and the maximum likelihood estimate does not exist.

Proof. Consider $β = t b$ for $t > 0$ . For an observation with $y_{i} = 1$ the linear predictor is $t x_{i}^{⊤} b \to + \infty$ , so $p_{i} \to 1$ and its log-likelihood contribution $lo g p_{i} \to 0$ . For an observation with $y_{i} = 0$ the predictor tends to $- \infty$ , so $p_{i} \to 0$ and the contribution $lo g (1 - p_{i}) \to 0$ .

Every term therefore increases toward its supremum of $0$ , so $ℓ (t b)$ increases to $0$ as $t \to \infty$ — the supremum of the log-likelihood is $0$ , and $0$ is not attained by any finite $β$ since that would require some $p_{i}$ to equal exactly $0$ or $1$ . Hence no maximiser exists, and the fitted coefficients diverge.∎

Pitfall. Separation is easy to create accidentally and is reported confusingly by software: coefficients of magnitude $20$ or more with standard errors in the hundreds, or a warning about fitted probabilities numerically $0$ or $1$ . It is not a convergence failure to be fixed by more iterations — the estimate genuinely does not exist.

It happens most often with a categorical predictor having a category in which every outcome is the same, and it becomes more likely with small samples or many predictors. The usual remedies are to merge categories, drop the offending predictor, or use a penalised fit — Firth's correction, or a ridge-style penalty, both of which make the penalised likelihood attain a finite maximum.

Remark (Checking linearity in the logit, and how much data is needed). Linearity in the logit is checked by grouping the data on a continuous predictor, computing the empirical log-odds within each group, and plotting them against the predictor: the plot should be roughly straight. Curvature is repaired as it would be in ordinary regression, by adding a quadratic term or splines.

For sample size, the binding constraint is not $n$ but the number of observations in the rarer outcome class. A common working rule asks for at least ten such observations per predictor, so a model with five predictors and a $5%$ event rate wants at least $50$ events, hence about $1000$ observations. Below that, coefficients are biased away from zero and the Wald standard errors are unreliable.

Example 8.20 (Checking linearity in the logit). A continuous predictor is grouped into five bins, giving success counts $3/40$ , $9/40$ , $20/40$ , $31/40$ , $36/40$ at bin midpoints $1, 2, 3, 4, 5$ . Assess linearity in the logit.

Solution.

Compute the empirical proportions: $0.075$ , $0.225$ , $0.500$ , $0.775$ , $0.900$ .
Convert to log-odds, $lo g {p / (1 - p)}$ :

lo g \frac{0.075}{0.925} = - 2.512, lo g \frac{0.225}{0.775} = - 1.237, 0, 1.237, 2.197.

The successive differences are $1.275$ , $1.237$ , $1.237$ , $0.960$ — close to constant across the middle and tailing off slightly at the top.
A straight line is a good description, so linearity in the logit is supported, and the common difference of about $1.2$ estimates the slope.
Sanity check on the raw proportions: their successive differences are $0.150, 0.275, 0.275, 0.125$ — markedly not constant, rising then falling. That is exactly the S-shape the logit transform is there to straighten, and it confirms that linearity should be assessed on the logit scale rather than the probability scale.
What would have indicated a problem: log-odds differences that changed systematically, say $2.0, 1.2, 0.6, 0.2$ , which would call for a quadratic term or a spline.

□

Example 8.21 (Diagnosing an implausible fit). A model predicting loan default from twelve predictors on $180$ loans, of which $14$ defaulted, reports one coefficient of $22.4$ with standard error $4, 812$ . What has happened?

Solution.

A coefficient of $22.4$ means an odds ratio of $e^{22.4} \approx 5.4 \times 1 0^{9}$ per unit — not a finding, an artefact.
The enormous standard error is the tell. Recall that $SE$ comes from $(X^{⊤} WX)^{- 1}$ with weights $p_{i} (1 - p_{i})$ ; as fitted probabilities approach $0$ or $1$ the weights approach zero, the information matrix becomes near-singular, and its inverse blows up.
This is separation, by the theorem Separation destroys the maximum likelihood estimate: some predictor (or combination) perfectly divides the defaulters from the rest, and the likelihood is being maximised by pushing that coefficient toward infinity. The iteration simply stopped at whatever the convergence tolerance allowed.
The sample size makes it likely. With $14$ events and $12$ predictors, the events-per-predictor ratio is $14/12 = 1.2$ , far below the working rule of $10$ — with that many dimensions and that few events, a perfect divider almost always exists.
Sanity check on the direction of the fix: more iterations will not help, since there is no finite maximum to converge to. Reduce the predictors to at most one or two, or fit with Firth's penalty, which guarantees finite estimates.

□

Assessing a fitted classifier

A logistic model returns probabilities. Turning those into decisions requires a threshold, and judging the result requires more than one number.

Definition 8.22 (Confusion matrix and its rates). Classifying as positive when $\overset{p}{^}_{i} > c$ produces counts of true positives $TP$ , false positives $FP$ , true negatives $TN$ and false negatives $FN$ . Define

sensitivity = \frac{TP}{TP + FN}, specificity = \frac{TN}{TN + FP}, precision = \frac{TP}{TP + FP} .

Proposition 8.23 (The threshold trades the two error rates). Sensitivity is non-increasing in the threshold $c$ and specificity is non-decreasing. At $c = 0$ sensitivity is $1$ and specificity $0$ ; at $c = 1$ the reverse.

Proof. Raising $c$ can only move cases from predicted-positive to predicted-negative, never the other way. So $TP$ cannot rise and $FN$ cannot fall, making sensitivity non-increasing; simultaneously $TN$ cannot fall and $FP$ cannot rise, making specificity non-decreasing. At $c = 0$ every case is called positive, so $FN = 0$ and $TN = 0$ , giving sensitivity $1$ and specificity $0$ ; at $c = 1$ every case is called negative.∎

Because the two rates move against each other, no threshold is "correct" without a statement of what the two errors cost. A screening test where a missed case is fatal and a false alarm means one more test should use a low threshold; a test triggering an invasive procedure should not.

Definition 8.24 (ROC curve and AUC). The ROC curve plots sensitivity against $1 -$ specificity as the threshold sweeps from $1$ to $0$ . The area under the curve (AUC) summarises it in one number, and equals the probability that a randomly chosen positive case receives a higher fitted probability than a randomly chosen negative one.

Figure 8.25. ROC curves for three models. The diagonal is chance: a model whose scores carry no information about the outcome traces it, with AUC

0.5

. The curve for the stronger model bows further toward the top-left corner, where sensitivity is high while the false-positive rate is still low. AUC measures the whole curve and so is independent of any particular threshold — useful for comparing models, and useless for choosing an operating point, which requires knowing what the two errors cost.

Example 8.26 (Reading a confusion matrix). At threshold $0.5$ , a model applied to $200$ cases gives $TP = 45$ , $FN = 15$ , $FP = 25$ , $TN = 115$ . Compute the rates, and say what changes if the threshold is raised to $0.7$ giving $TP = 32$ , $FN = 28$ , $FP = 8$ , $TN = 132$ .

Solution.

At $c = 0.5$ : sensitivity $= 45/60 = 0.75$ ; specificity $= 115/140 = 0.8214$ ; precision $= 45/70 = 0.6429$ ; accuracy $= (45 + 115) /200 = 0.80$ .
At $c = 0.7$ : sensitivity $= 32/60 = 0.5333$ ; specificity $= 132/140 = 0.9429$ ; precision $= 32/40 = 0.80$ ; accuracy $= 164/200 = 0.82$ .
Raising the threshold cut sensitivity by $0.217$ and raised specificity by $0.121$ — the trade of the proposition The threshold trades the two error rates, in both required directions ✓.
Accuracy rose slightly, from $0.80$ to $0.82$ , which illustrates why accuracy is a poor summary: it improved while the model became much worse at finding the cases it exists to find.
Sanity check on the prevalence: only $60$ of $200$ cases are positive, so predicting "negative" for everything would score $140/200 = 0.70$ accuracy while detecting nothing at all. Any accuracy figure must be read against that baseline.

□

Pitfall. Accuracy is the wrong metric for an imbalanced outcome. With a $1%$ event rate, a model that never predicts the event is $99%$ accurate and worthless. Report sensitivity and specificity, or precision and recall, and choose the threshold from the relative cost of the two errors.

A second caution. AUC and calibration are different properties. A model whose fitted probabilities are all half their correct values ranks cases perfectly — so its AUC is unchanged — while every probability it reports is wrong. If the probabilities themselves will be used, check calibration by grouping cases into bins of predicted probability and comparing the mean prediction in each bin with the observed proportion; the pairs should lie near the diagonal.

Exercises for §8.4 →

8.5Generalised linear models

Logistic regression is not a special technique but an instance of a pattern, and naming the pattern makes several other models immediate.

Definition 8.27 (Generalised linear model). A generalised linear model specifies three things: a distribution for the response from the exponential family, a linear predictor $η_{i} = x_{i}^{⊤} β$ , and a link function $g$ connecting them through

g (E (Y_{i})) = η_{i} .

Ordinary linear regression is the choice (normal, identity link); logistic regression is (Bernoulli, logit link). Two more choices cover most of what remains.

Response	Distribution	Link	Model
continuous	normal	identity	linear regression
binary	Bernoulli	logit	logistic regression
count	Poisson	log	Poisson regression
count, over-dispersed	negative binomial	log	negative binomial regression

Proposition 8.28 (Poisson regression coefficients are log rate ratios). In the model $lo g E (Y_{i}) = x_{i}^{⊤} β$ with $Y_{i}$ Poisson, increasing $x_{j}$ by one multiplies the expected count by $e^{β_{j}}$ .

Proof. $E (Y) = e^{η}$ , so replacing $η$ by $η + β_{j}$ multiplies the mean by $e^{β_{j}}$ , independently of the starting value — the same argument as the theorem Coefficients are log odds ratios, with the log link in place of the logit.∎

Remark (Why the log link for counts, and what over-dispersion means). The log link is chosen for counts for the same reason the logit was chosen for probabilities: it maps the constrained range of the mean — here $(0, \infty)$ — onto the whole real line, so no combination of predictors can produce an impossible negative expected count.

The Poisson model carries a strong implication: by the properties of the Poisson distribution the variance equals the mean. Real count data frequently has variance well above the mean, a condition called over-dispersion, which leaves the coefficients roughly unbiased but makes the standard errors too small — the same pattern as heteroscedasticity in the linear model. The negative binomial distribution adds a dispersion parameter and is the standard remedy.

Remark (More than two categories). Two extensions handle responses with $k > 2$ categories.

Multinomial logistic regression picks a reference category and fits $k - 1$ separate logit equations, each comparing one category against the reference — exactly the indicator coding of the ANOVA chapter, applied to the response instead of a predictor. It makes no assumption about ordering, and costs $k - 1$ coefficients per predictor.

Ordinal logistic regression exploits an ordering when there is one (a rating from "poor" to "excellent"). The usual form, the proportional odds model, fits $k - 1$ intercepts but a single slope per predictor, assuming the effect on the log-odds of being above any threshold is the same at every threshold. That assumption buys a great deal of parsimony and should be checked rather than assumed.

Example 8.29 (Choosing a model). For each response, name the appropriate GLM: (a) number of customer complaints per store per month; (b) whether a subscriber renews; (c) a satisfaction rating from 1 to 5; (d) time in minutes spent on a page.

Solution.

(a) A count with no fixed upper bound: Poisson regression with a log link. Check for over-dispersion; if the variance much exceeds the mean, use negative binomial.
(b) Binary: logistic regression.
(c) Ordered categories: ordinal logistic regression, which respects the ordering and spends one slope per predictor rather than four. Multinomial would also fit but discards the ordering and estimates four times as many coefficients.
(d) Continuous and positive, and likely right-skewed: ordinary regression on the log of the time is the simplest option, or a gamma GLM with a log link if the skew is severe.
Sanity check on the pattern: in every case the link is chosen so that the linear predictor, which ranges over all of $R$ , maps onto the range the mean is allowed to occupy. That single requirement determines the identity link for an unconstrained mean, the log link for a positive one, and the logit for one confined to $(0, 1)$ .

□

Exercises for §8.5 →

Common Mistakes

Reading a coefficient as a change in probability. It is a change in log-odds; the probability effect depends on where you start and is largest near $p = \frac{1}{2}$ .
Reporting an odds ratio as a relative risk. They agree only for rare outcomes; at $p = 0.4 \to 0.6$ the odds ratio is $2.25$ against a relative risk of $1.5$ .
Exponentiating a confidence interval the wrong way round. Build the interval on the log-odds scale, where the estimate is approximately normal, then transform the endpoints.
Fitting a linear probability model and ignoring predictions outside $[0, 1]$ . Heteroscedasticity can be patched; an impossible probability cannot.
Treating separation as a convergence problem. The maximum likelihood estimate does not exist; more iterations cannot find it. Use a penalised fit or fewer predictors.
Counting $n$ rather than events. The binding constraint is the number of observations in the rarer class, with about ten per predictor as a working minimum.
Assuming Poisson regression fits any count. It forces the variance to equal the mean; over-dispersed data needs a negative binomial or a robust standard error.
Using multinomial regression for an ordered response. It works, but discards the ordering and multiplies the number of coefficients.

About the practice questions. They check that you can carry out this chapter's computations correctly, and each one is graded on a single answer. They are not proof exercises: working through them confirms the mechanics, not that you could prove the results yourself. For that, re-read the statements above and try to reconstruct their proofs with the page closed. This chapter also has a printable problem set: proofs, sketches and find-the-error work to do on paper, with full solutions on a separate page.

Practice this chapter Printable problem set