Matrices

An $m \times n$ matrix has $m$ rows and $n$ columns. The entry in row $i$, column $j$ is $A_{ij}$. A square matrix has $m = n$.

Matrix addition: $(A+B)_{ij} = A_{ij}+B_{ij}$. Both matrices must have the same dimensions; you cannot add a $3\times 2$ to a $2\times 3$.

Scalar multiplication: $(cA)_{ij} = c\,A_{ij}$ — every entry is scaled by $c$.

The transpose $A^T$ is obtained by swapping rows and columns: $(A^T)_{ij} = A_{ji}$. If $A$ is $m\times n$, then $A^T$ is $n\times m$. Key properties: $(AB)^T = B^TA^T$ (order reverses), $(A^T)^T = A$, $(A+B)^T = A^T+B^T$.

A matrix is symmetric if $A = A^T$. Symmetric matrices appear ubiquitously in statistics (covariance matrices), physics (inertia tensors), and optimisation (Hessians). They are always square, and their eigenvalues are always real.

The trace of a square matrix is the sum of its diagonal entries: $\text{tr}(A) = \sum_{i=1}^n A_{ii}$. It equals the sum of the eigenvalues of $A$ and satisfies $\text{tr}(AB) = \text{tr}(BA)$ even when $AB \neq BA$.

If $A$ is $m\times n$ and $B$ is $n\times p$, the product $AB$ is $m\times p$. The inner dimensions must match: columns of $A$ must equal rows of $B$.

Each entry is a dot product: $(AB)_{ij} = \sum_{k=1}^{n} A_{ik}\,B_{kj}$ — row $i$ of $A$ dotted with column $j$ of $B$.

Why this definition? It encodes composition of linear transformations. If $T_A: \mathbb{R}^n\to\mathbb{R}^m$ and $T_B: \mathbb{R}^p\to\mathbb{R}^n$, then the composed map $T_A \circ T_B$ is represented by $AB$. The rule $(AB)\mathbf{x} = A(B\mathbf{x})$ forces this definition.

Matrix multiplication is not commutative: $AB \neq BA$ in general, even when both products exist and have the same size. It is associative: $(AB)C = A(BC)$, and distributive: $A(B+C) = AB+AC$.

The identity matrix $I_n$ (ones on the diagonal, zeros elsewhere) satisfies $AI = IA = A$. It is the matrix analogue of the number $1$.

Column picture: $A\mathbf{v} = v_1\mathbf{a}_1 + v_2\mathbf{a}_2 + \cdots + v_n\mathbf{a}_n$, a linear combination of the columns of $A$ with coefficients from $\mathbf{v}$. This column view is the key to understanding column spaces and linear independence.

The determinant is a scalar assigned to every square matrix that captures whether it is invertible, how it scales area/volume, and whether it preserves orientation.

For $2\times 2$: $\det\begin{pmatrix}a&b\\c&d\end{pmatrix} = ad - bc$. The main-diagonal product minus the anti-diagonal product.

Geometric meaning: $|\det(A)|$ is the factor by which the transformation $A$ multiplies area ($n=2$) or volume ($n=3$). $\det(A)=2$ means areas are doubled; $\det(A)=0$ means the transformation collapses space to a lower dimension.

Sign of the determinant: $\det(A)>0$ means orientation is preserved (no reflection). $\det(A)<0$ means orientation is reversed.

The invertibility theorem: $A$ is invertible $\iff$ $\det(A) \neq 0$. A singular matrix (det $=0$) squashes space and cannot be undone — its inverse does not exist. For $3\times 3$ and larger matrices, cofactor expansion is covered in Chapter 6.

The inverse $A^{-1}$ of a square matrix $A$ satisfies $AA^{-1} = A^{-1}A = I$. It exists if and only if $\det(A) \neq 0$. A matrix without an inverse is called singular.

For $2\times 2$: $A^{-1} = \frac{1}{ad-bc}\begin{pmatrix}d&-b\\-c&a\end{pmatrix}$. Swap the main-diagonal entries, negate the off-diagonal entries, and divide by the determinant.

For larger matrices, use Gauss-Jordan elimination: row-reduce the augmented matrix $[A\;|\;I]$. When the left block reaches $I$, the right block is $A^{-1}$.

The inverse solves $A\mathbf{x}=\mathbf{b}$ immediately: $\mathbf{x} = A^{-1}\mathbf{b}$. However, computing $A^{-1}$ explicitly is costly. Gaussian elimination solves systems more efficiently without forming the inverse.

Key properties: $(AB)^{-1} = B^{-1}A^{-1}$ (order reverses), $(A^{-1})^{-1}=A$, $(A^T)^{-1}=(A^{-1})^T$, $\det(A^{-1})=1/\det(A)$.

Orthogonal matrices satisfy $Q^TQ=I$, so $Q^{-1}=Q^T$. For these matrices (rotations and reflections), the inverse is free — just transpose.

Symmetric matrices: $A = A^T$. By the Spectral Theorem (Chapter 8), all eigenvalues are real and there exists an orthonormal eigenvector basis. These arise in covariance matrices, Hessians, and physics.

Orthogonal matrices: $Q^TQ = I$. Columns are orthonormal. $\det(Q) = \pm 1$. They represent rotations ($\det=+1$) and reflections ($\det=-1$). The inverse is simply the transpose: $Q^{-1}=Q^T$.

Diagonal matrices: nonzero entries only on the main diagonal. Trivial to invert ($D^{-1}_{ii}=1/D_{ii}$), multiply ($(D_1D_2)_{ii}=(D_1)_{ii}(D_2)_{ii}$), and exponentiate. Diagonalisation is the act of finding a basis in which a matrix looks diagonal.

Upper/lower triangular: all entries below/above the diagonal are zero. The determinant equals the product of diagonal entries. Systems $L\mathbf{x}=\mathbf{b}$ (lower) are solved by forward substitution; $U\mathbf{x}=\mathbf{b}$ (upper) by back substitution. The LU decomposition writes any invertible matrix as $A=LU$.

Idempotent matrices: $P^2=P$. Projection matrices are idempotent — projecting twice gives the same result. They have eigenvalues $0$ and $1$ only.