Statistical Inference: Hypothesis Testing

The null hypothesis $H_0$ is the default claim (no effect, no difference, status quo). The alternative $H_a$ is the claim the investigator wants to support. We begin by assuming $H_0$ is true and ask: how surprising is the observed data under this assumption?

The test statistic measures how many standard errors the sample result is from the value specified by $H_0$. For a mean: $z = (\bar{x}-\mu_0)/(\sigma/\sqrt{n})$ (if $\sigma$ known) or $t = (\bar{x}-\mu_0)/(s/\sqrt{n})$ (if $\sigma$ estimated).

The $p$-value is $P(X \geq x_\text{obs} \mid H_0)$ — the probability of observing data at least as extreme as yours, assuming $H_0$ is true. Small $p$-values ($p \leq \alpha$) lead to rejection of $H_0$; large $p$-values do not.

One-sided vs. two-sided tests: $H_a: \mu > \mu_0$ is right-sided (one-tailed, $p = P(Z > z_\text{obs})$); $H_a: \mu < \mu_0$ is left-sided; $H_a: \mu \neq \mu_0$ is two-sided ($p = 2P(Z > |z_\text{obs}|)$). Choose the direction of $H_a$ before collecting data.

Equivalence of tests and intervals: reject $H_0: \mu = \mu_0$ at level $\alpha$ (two-sided) if and only if the $(1-\alpha)\times 100\%$ confidence interval for $\mu$ does not contain $\mu_0$. Confidence intervals and tests are dual procedures.

Type I error ($\alpha$): rejecting $H_0$ when it is true (false positive). By choosing $\alpha$ you directly control the probability of this error. Common choices are $\alpha = 0.05$, $0.01$, or $0.10$ depending on the consequences of false rejection.

Type II error ($\beta$): failing to reject $H_0$ when it is false (false negative). $\beta$ is not directly controlled by $\alpha$ — it depends on $n$, the true effect size, and $\sigma$.

Power $= 1 - \beta$: the probability of correctly detecting a real effect when it exists. Power increases with larger $n$, larger true effect size, larger $\alpha$, and smaller $\sigma$. The conventional target is $80\%$ power.

The $\alpha$-$\beta$ trade-off: for fixed $n$, lowering $\alpha$ (stricter rejection) increases $\beta$ (more missed effects). Increasing $n$ is the only way to improve power without inflating $\alpha$.

Sample size calculation: for a one-sample $z$-test at significance $\alpha$ and power $1-\beta$, $n = (z_{\alpha/2}+z_\beta)^2\sigma^2/\delta^2$, where $\delta = \mu_a - \mu_0$ is the detectable difference.

One-sample $t$-test: $H_0: \mu = \mu_0$, test statistic $t = (\bar{x}-\mu_0)/(s/\sqrt{n}) \sim t_{n-1}$ under $H_0$. Used when $\sigma$ is unknown — which is essentially always.

Two-sample (independent) $t$-test: compares $\mu_1$ and $\mu_2$. Test statistic $t = (\bar{x}_1-\bar{x}_2)/\sqrt{s_1^2/n_1+s_2^2/n_2}$ (Welch's version, does not assume equal variances). Degrees of freedom estimated by the Welch-Satterthwaite formula.

Paired $t$-test: for matched pairs $(x_i, y_i)$, compute differences $d_i = x_i - y_i$ and apply a one-sample $t$-test on $d_i$. More powerful than two-sample when pairs are correlated — it removes between-subject variability.

The $z$-test uses $\sigma$ known; the $t$-test estimates $\sigma$ with $s$. For $n \geq 30$ the difference is negligible in practice. For small $n$, the heavier tails of the $t$-distribution properly account for the additional uncertainty in estimating $\sigma$.

The chi-square goodness-of-fit test checks whether observed category counts match a hypothesised distribution. $\chi^2 = \sum (O_i - E_i)^2/E_i$ with $k-1$ degrees of freedom ($k$ = number of categories).

The one-sample proportion $z$-test: $H_0: p = p_0$, test statistic $z = (\hat{p}-p_0)/\sqrt{p_0(1-p_0)/n}$. Valid when $np_0 \geq 10$ and $n(1-p_0) \geq 10$.

Two-proportion $z$-test: $z = (\hat{p}_1-\hat{p}_2)/SE$, where $SE = \sqrt{\hat{p}(1-\hat{p})(1/n_1+1/n_2)}$ using the pooled estimate $\hat{p} = (x_1+x_2)/(n_1+n_2)$ under $H_0: p_1=p_2$.

If you run $m$ independent tests each at level $\alpha$, the probability of at least one false positive is $1 - (1-\alpha)^m$. For $m=20$ and $\alpha=0.05$, this is $1 - 0.95^{20} \approx 0.64$ — a $64\%$ chance of at least one spurious significant result.

The Bonferroni correction: to keep the family-wise error rate at $\alpha$, use $\alpha/m$ for each individual test. It is conservative (too strict when tests are positively correlated).

The False Discovery Rate (FDR): the Benjamini-Hochberg procedure controls the expected fraction of false rejections among all rejections. Less conservative than Bonferroni, preferred for exploratory work.

p-hacking: running many analyses and selectively reporting significant ones. It inflates the Type I error rate and is a major cause of the replication crisis in science.