Sampling and Data Distributions

A population is the complete set of individuals or observations of interest. A parameter (e.g., $\mu$, $\sigma^2$, $p$) is a fixed numerical characteristic of the population — unknown and usually unknowable without a census.

A sample is a subset of the population. A statistic (e.g., $\bar{x}$, $s^2$, $\hat{p}$) is computed from sample data. Statistics are observable but vary from sample to sample — they are random variables.

A statistic $\hat{\theta}$ is an unbiased estimator of parameter $\theta$ if $E(\hat{\theta}) = \theta$. The sample mean $\bar{X}$ is unbiased for $\mu$. The sample variance $s^2 = \frac{1}{n-1}\sum(x_i-\bar{x})^2$ is unbiased for $\sigma^2$.

Sampling design: simple random sampling (SRS) gives every sample of size $n$ equal probability. Stratified sampling divides the population into groups (strata) and draws SRSs from each, improving precision. Cluster sampling and systematic sampling are used when SRS is impractical.

Bias in sampling: a convenience sample (whoever is available) or voluntary response sample systematically misrepresents the population. No amount of statistical analysis can fix a biased design — always randomise.

Weak Law of Large Numbers: for i.i.d. random variables $X_1, X_2, \ldots$ with mean $\mu$, the sample mean $\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$ converges in probability to $\mu$: for any $\varepsilon > 0$, $P(|\bar{X}_n - \mu| > \varepsilon) \to 0$ as $n \to \infty$.

The Strong Law of Large Numbers guarantees almost sure convergence: $P(\bar{X}_n \to \mu) = 1$. Almost every sequence of i.i.d. observations has a time-average that converges to the true mean.

The LLN justifies using observed frequencies as probability estimates, trusting large studies over small ones, and the general principle that more data gives better estimates.

The LLN does not imply the 'gambler's fallacy.' After $10$ coin flips all landing heads, the next flip is still $50/50$. The LLN talks about the eventual average, not compensation of past outcomes.

Central Limit Theorem (CLT): if $X_1, \ldots, X_n$ are i.i.d. with mean $\mu$ and finite variance $\sigma^2$, then the standardised mean $(\bar{X}-\mu)/(\sigma/\sqrt{n}) \to N(0,1)$ in distribution as $n\to\infty$. Equivalently, $\bar{X} \approx N(\mu, \sigma^2/n)$ for large $n$.

This is remarkable because the result holds regardless of the population's shape — whether it is skewed, bimodal, or uniform. The averaging process irons out all non-normality.

The standard error of the mean is $SE = \sigma/\sqrt{n}$. Doubling $n$ reduces $SE$ by a factor of $\sqrt{2}$, not $2$. To halve the standard error you need to quadruple the sample size.

Rule of thumb: $n \geq 30$ is often sufficient for the normal approximation to be good, but heavily skewed populations may require $n \geq 100$ or more. Always check with a histogram of the data.

The CLT also applies to sums: $S_n = \sum X_i \approx N(n\mu, n\sigma^2)$. And to proportions: $\hat{p} = X/n \approx N(p, p(1-p)/n)$ when $np \geq 10$ and $n(1-p) \geq 10$.

The sample proportion $\hat{p} = X/n$ (where $X$ counts successes in $n$ Bernoulli trials) estimates the population proportion $p$.

Mean: $E(\hat{p}) = p$ (unbiased). Standard error: $SE(\hat{p}) = \sqrt{p(1-p)/n}$.

By the CLT: $\hat{p} \approx N(p, p(1-p)/n)$ when $np \geq 10$ and $n(1-p) \geq 10$.

The sampling distribution of the difference $\hat{p}_1 - \hat{p}_2$ (from two independent samples) is approximately $N(p_1-p_2, p_1(1-p_1)/n_1 + p_2(1-p_2)/n_2)$. This is used for two-proportion $z$-tests and $z$-intervals.