Chapter 2 · 16 min

2.7 — Backprop, part 3: weight and bias gradients

Goal: Express the gradient of the loss w.r.t. each layer's weights and biases in terms of the layer's delta and its input activation.

1. Why this session exists

Sessions 2.5 and 2.6 gave us $\delta^{(\ell)}$ at every layer. The last step of backprop is to convert these deltas into the gradients we update with: $\partial L / \partial W^{(\ell)}$ and $\partial L / \partial b^{(\ell)}$ . The conversion is one line each, and the pattern is the same "residual × input" outer product we saw in 1.3 and 1.8 — just applied at every layer of the network.[1]

2. Per-example formula

At layer $\ell$ , the pre-activation is

z^{(\ell)} \;=\; W^{(\ell)} a^{(\ell-1)} + b^{(\ell)}.

Entry by entry, $z^{(\ell)}_i = \sum_j W^{(\ell)}_{i, j}\, a^{(\ell-1)}_j + b^{(\ell)}_i$ . So

\frac{\partial z^{(\ell)}_i}{\partial W^{(\ell)}_{i, j}} = a^{(\ell-1)}_j, \qquad \frac{\partial z^{(\ell)}_p}{\partial W^{(\ell)}_{i, j}} = 0 \text{ for } p \ne i.

By the chain rule with $\delta^{(\ell)} = \partial L / \partial z^{(\ell)}$ ,

\frac{\partial L}{\partial W^{(\ell)}_{i, j}} \;=\; \sum_p \frac{\partial L}{\partial z^{(\ell)}_p} \cdot \frac{\partial z^{(\ell)}_p}{\partial W^{(\ell)}_{i, j}} \;=\; \delta^{(\ell)}_i \cdot a^{(\ell-1)}_j.

Stacking entries $(i, j)$ into a matrix,

\boxed{\ \frac{\partial L}{\partial W^{(\ell)}} \;=\; \delta^{(\ell)}\, \bigl(a^{(\ell-1)}\bigr)^\top \;\in\; \mathbb{R}^{d_\ell \times d_{\ell-1}}.\ }

Outer product: delta column at layer $\ell$ times activation row at layer $\ell - 1$ .

For the bias, $\partial z^{(\ell)}_p / \partial b^{(\ell)}_i = \delta_{pi}$ , so

\boxed{\ \frac{\partial L}{\partial b^{(\ell)}} \;=\; \delta^{(\ell)} \;\in\; \mathbb{R}^{d_\ell}.\ }

3. Batched form

For a mini-batch of $B$ examples, with $\Delta^{(\ell)} \in \mathbb{R}^{B \times d_\ell}$ and $A^{(\ell-1)} \in \mathbb{R}^{B \times d_{\ell-1}}$ as before, the weight gradient is the sum of per-example outer products, scaled by $1 / B$ :

\boxed{\ \frac{\partial L_{\text{batch}}}{\partial W^{(\ell)}} \;=\; \frac{1}{B}\, (\Delta^{(\ell)})^\top\, A^{(\ell-1)} \;\in\; \mathbb{R}^{d_\ell \times d_{\ell-1}}.\ }

(Recover the per-example case by inspecting one row of $\Delta$ and one row of $A$ ; the matrix product over $B$ rows is the sum of outer products.)

The bias gradient is the column-wise average of $\Delta^{(\ell)}$ :

\boxed{\ \frac{\partial L_{\text{batch}}}{\partial b^{(\ell)}} \;=\; \frac{1}{B}\, \sum_{i=1}^B \Delta^{(\ell)}_{i, :} \;\in\; \mathbb{R}^{d_\ell}.\ }

In PyTorch this is Delta.T @ A / B for the weight and Delta.mean(dim=0) for the bias.

4. Shape sanity check

$W^{(\ell)}$ is $d_\ell \times d_{\ell-1}$ .
$\Delta^{(\ell)}$ is $B \times d_\ell$ , $A^{(\ell-1)}$ is $B \times d_{\ell-1}$ .
$(\Delta^{(\ell)})^\top A^{(\ell-1)}$ is $d_\ell \times d_{\ell-1}$ ✓ (matches $W^{(\ell)}$ ).
$\Delta^{(\ell)}$ summed over rows is $d_\ell$ ✓ (matches $b^{(\ell)}$ ).

A gradient must be the same shape as the parameter it differentiates. If you ever see a transposed result, you've swapped the order in the outer product.

5. The full forward + backward picture

End-to-end picture of one training step on a mini-batch.

Forward (vectorised): for $\ell = 1, \dots, L$ ,

Z^{(\ell)} = A^{(\ell-1)} (W^{(\ell)})^\top + b^{(\ell)}, \qquad A^{(\ell)} = g^{(\ell)}(Z^{(\ell)}).

Compute $L_{\text{batch}}$ .

Backward (vectorised):

Initialise $\Delta^{(L)}$ at the output layer (2.5; for softmax + CE it is $\tfrac{1}{B}(\hat Y - Y)$ ).
For $\ell = L - 1, L - 2, \dots, 1$ , recurse $\Delta^{(\ell)} = (\Delta^{(\ell+1)} W^{(\ell+1)}) \odot g'^{(\ell)}(Z^{(\ell)})$ (2.6).
For each $\ell$ , compute $\partial L / \partial W^{(\ell)} = (\Delta^{(\ell)})^\top A^{(\ell-1)}$ and $\partial L / \partial b^{(\ell)} = \Delta^{(\ell)}\, \text{sum over batch axis}$ .

(The $1/B$ factor lives wherever you decide; standard practice folds it into the output delta or the loss itself.)

Update. For each layer,

W^{(\ell)} \leftarrow W^{(\ell)} - \alpha\, \frac{\partial L}{\partial W^{(\ell)}}, \qquad b^{(\ell)} \leftarrow b^{(\ell)} - \alpha\, \frac{\partial L}{\partial b^{(\ell)}}.

One training step ≈ three forward passes' worth of FLOPs. Convergence is the topic of Chapter 3.

6. Takeaways

$\partial L / \partial W^{(\ell)} = \delta^{(\ell)} (a^{(\ell-1)})^\top$ . Outer product: delta × input.
$\partial L / \partial b^{(\ell)} = \delta^{(\ell)}$ .
Batched: $\partial L / \partial W^{(\ell)} = \tfrac{1}{B} (\Delta^{(\ell)})^\top A^{(\ell-1)}$ , bias is the batch-mean of $\Delta^{(\ell)}$ .
Combined with 2.5 and 2.6, you now have the complete backprop algorithm.

Self-test

(a) Derive the weight gradient identity from scratch using the chain rule on $z^{(\ell)} = W^{(\ell)} a^{(\ell-1)} + b^{(\ell)}$ . Check that the result is an outer product.

(b) For the MNIST MLP of 2.4 §4 (widths $(784, 256, 128, 10)$ ) and batch $B = 32$ , compute the FLOP count of one forward pass and of the corresponding backward pass. Verify backward ≈ forward.

(c) Suppose two examples in the same mini-batch have very different deltas — one $\delta = +5$ , another $\delta = -5$ . How does the mean gradient over the batch behave? Why is this a feature, not a bug? (Hint: noise averaging in SGD.)

(d) For layer 1, the input is the data itself: $a^{(0)} = x$ . Write out $\partial L / \partial W^{(1)}$ explicitly in terms of $\delta^{(1)}$ and the raw input.

References

[1]
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors". Nature, 323(6088), 533–536. — Backpropagation (2.5–2.7).
↩