0.2 — Matrix calculus: derivatives that matter for backprop

1. Convention: numerator vs. denominator

Before any derivative, fix the convention. For $f : \mathbb{R}^n \to \mathbb{R}^m$ given by $y = f(x)$, numerator layout declares

Denominator layout flips it: $\partial y / \partial x \in \mathbb{R}^{n \times m}$. I pick numerator because it makes the chain rule stack matrices in the natural multiplication order. Every derivation in this course uses numerator layout. Mix them and you suffer.[1]

2. Derivative of $Wx + b$ with respect to $x$

Let $f(x) = Wx + b$ with $W \in \mathbb{R}^{m\times n}$, $b \in \mathbb{R}^m$. Component-wise,

Differentiating $y_i$ with respect to $x_j$, every term with $k \neq j$ vanishes:

Stacking over every $(i, j)$ pair,

The intuition is direct: differentiating an affine map with respect to its input recovers its linear part.

3. Derivative with respect to $W$

Here $W$ is a matrix, so $\partial y / \partial W$ is in principle a rank-3 tensor: it has indices $(i, j, k)$ with $i$ ranging over $y$ and $(j, k)$ ranging over $W$. The cleanest route is to differentiate index by index; the storage shape will fall out at the end.

Note the $\delta_{ij}$: the derivative of the $i$-th output with respect to the element $W_{jk}$ is non-zero only when $j = i$. In other words, the weight $W_{jk}$ affects only the output $y_j$.

4. Why this shows up as an outer product in backprop

At training time, what we need is not the full tensor but the gradient of a scalar loss $L$ with respect to $W$. By the chain rule:

Define $\delta := \partial L / \partial y \in \mathbb{R}^{m}$. In matrix form, $\partial L / \partial W$ is the $m \times n$ matrix whose $(j,k)$ entry is $\delta_j\, x_k$ — the outer product:

And for the bias, $\partial L/\partial b = \delta$ (each $b_i$ contributes $1$ to $y_i$). This completes the local update rule of a linear layer; chained across all layers, it is PyTorch's backward().

5. Quick check (do it with a pen)

For $W = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}$ and $x = (5, 6)^\top$, compute $y = Wx$, then $\partial y / \partial x$, and finally — assuming $L = \tfrac{1}{2}\|y\|^2$ — obtain $\delta$ and the outer product $\delta\, x^\top$. If $W^\top y\, x^\top$ shows up in the slot where $\delta\, x^\top$ should be, the numerator convention from §1 was crossed.