Statistical Inference: Estimation

A sampling distribution is the probability distribution of a statistic (such as $\bar{X}$) computed from all possible random samples of size $n$ from a population. It describes how the statistic varies from sample to sample.

The standard error of the mean is $SE_{\bar{X}} = \sigma/\sqrt{n}$, or estimated as $s/\sqrt{n}$ when $\sigma$ is unknown. It quantifies the precision of $\bar{X}$ as an estimator of $\mu$ — larger $n$ gives smaller $SE$ and therefore more precise estimates.

By the Central Limit Theorem, $\bar{X}$ is approximately $N(\mu, \sigma^2/n)$ for large $n$. This holds regardless of the shape of the population distribution, making normal-based inference broadly applicable.

A statistic is an unbiased estimator of a parameter $\theta$ if $E(\hat{\theta}) = \theta$. The sample mean $\bar{X}$ is unbiased for $\mu$. The sample variance $s^2$ (with $n-1$) is unbiased for $\sigma^2$.

Mean Squared Error: $\text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + [\text{Bias}(\hat{\theta})]^2$. A good estimator minimises MSE — there is a bias-variance trade-off.

A point estimate is a single numerical value used as a best guess for a population parameter. Common examples: $\bar{x}$ estimates $\mu$; $s^2$ estimates $\sigma^2$; $\hat{p} = x/n$ estimates $p$.

Maximum Likelihood Estimation (MLE) finds the parameter value $\hat{\theta}$ that maximises the likelihood function $L(\theta) = \prod_{i=1}^n f(x_i;\theta)$ — the probability of observing the actual data, viewed as a function of $\theta$.

In practice, we maximise $\ell(\theta) = \ln L(\theta)$ (log-likelihood) which is easier to differentiate. Set $d\ell/d\theta = 0$ and solve. For a normal population, MLE gives $\hat{\mu}=\bar{x}$ and $\hat{\sigma}^2 = \frac{1}{n}\sum(x_i-\bar{x})^2$ (note: MLE uses $n$, not $n-1$).

MLE has excellent asymptotic properties: it is consistent, asymptotically efficient (achieves the Cramér-Rao lower bound), and asymptotically normal. For large samples it is the go-to estimation method.

A $100(1-\alpha)\%$ confidence interval provides a range of plausible values for an unknown parameter: $\bar{x} \pm z_{\alpha/2}\cdot SE$ for a known $\sigma$, or $\bar{x} \pm t_{\alpha/2,n-1}\cdot(s/\sqrt{n})$ when $\sigma$ is estimated.

Correct interpretation: if we were to repeat the sampling procedure many times, approximately $100(1-\alpha)\%$ of the resulting intervals would contain the true parameter. The interval either contains $\mu$ or it does not — after construction, there is no probability to speak of.

The margin of error $E = z^*\cdot SE$ determines the half-width of the interval. To achieve a desired margin of error $E$, use $n = (z^*\sigma/E)^2$.

The $t$-distribution has heavier tails than the normal and is appropriate when $\sigma$ is estimated from data. As degrees of freedom $df = n-1$ increases, the $t$-distribution converges to the standard normal. For $n \geq 30$ the difference is minimal in practice.

For a proportion $p$: $\hat{p} \pm z^*\sqrt{\hat{p}(1-\hat{p})/n}$. Valid when $n\hat{p}\geq 10$ and $n(1-\hat{p})\geq 10$.

A Type I error (false positive, size $\alpha$) occurs when $H_0$ is true but we reject it. We control this by choosing $\alpha$ before testing.

A Type II error (false negative, probability $\beta$) occurs when $H_0$ is false but we fail to reject it. Power $= 1-\beta$ is the probability of correctly detecting a real effect.

Factors that increase power: larger sample size $n$ (reduces $SE$, making it easier to detect real differences); larger true effect size (bigger signal); larger $\alpha$ (less strict rejection criterion); smaller $\sigma$ (less variability).

Sample size planning: choose $n$ to achieve a desired power (typically $80\%$ or $90\%$) at a specified effect size and significance level $\alpha$.

Statistical significance ($p \leq \alpha$) means the data is unlikely under $H_0$. Practical significance means the effect is large enough to matter in the real world. These are independent: large samples make tiny effects statistically significant; small samples may miss large effects.

Effect size measures the magnitude of an effect independently of sample size. Cohen's $d = (\bar{x}-\mu_0)/s$ for means; $r$ or $r^2$ for correlations; odds ratio for proportions. Always report effect size alongside $p$-values.

Confidence intervals are more informative than $p$-values alone: they show both the direction, magnitude, and uncertainty of an estimate — not merely a binary decision.