2.8 — Backprop by hand on a tiny network

1. Why this session exists

Sessions 2.5–2.7 derived the three identities. This one runs them. Take a 2-input, 2-hidden, 1-output network with sigmoid activations, pick a single example, and compute every $z$, $a$, $\delta$, $\partial L / \partial W$, $\partial L / \partial b$ by hand. Once the numbers come out right, the abstraction is yours.

This is the kind of exercise to do with a calculator and a notebook the first time, then re-check by writing a 15-line PyTorch script that backprops the same example and matches your numbers to 4 decimals.

2. The network

Widths $(d_0, d_1, d_2) = (2, 2, 1)$. Sigmoid at both the hidden and output layers. Binary cross-entropy loss. One example.

Parameters (chosen for clean arithmetic):

One example: $x = (1.0,\, 2.0)$, label $y = 1$.

3. Forward pass

Layer 1 pre-activation:

Layer 1 activation:

(Using $\sigma(0.50) = 1 / (1 + e^{-0.50}) \approx 0.6225$, and similarly $\sigma(1.10) \approx 0.7503$.)

Output pre-activation:

Output activation (prediction):

Loss (BCE, label $y = 1$):

4. Backward pass

Output delta (sigmoid + BCE, from 2.5):

Layer-2 gradients (a $1 \times 2$ row of weights and a scalar bias, from 2.7):

Layer-1 delta (recursion from 2.6):

First, $(W^{(2)})^\top \delta^{(2)}$:

And $\sigma'(z^{(1)}) = a^{(1)} \odot (1 - a^{(1)})$:

Hadamard product:

Layer-1 gradients:

5. The first SGD update

With learning rate $\alpha = 0.1$,

6. Verification in PyTorch

The following PyTorch snippet should reproduce the numbers above to 4 decimals.

import torch
import torch.nn as nn

torch.set_default_dtype(torch.float64)

x = torch.tensor([[1.0, 2.0]])
y = torch.tensor([[1.0]])

W1 = torch.tensor([[0.10, 0.20], [0.30, 0.40]], requires_grad=True)
b1 = torch.tensor([0.0, 0.0], requires_grad=True)
W2 = torch.tensor([[0.50, -0.60]], requires_grad=True)
b2 = torch.tensor([0.10], requires_grad=True)

z1 = x @ W1.T + b1
a1 = torch.sigmoid(z1)
z2 = a1 @ W2.T + b2
y_hat = torch.sigmoid(z2)
L = nn.functional.binary_cross_entropy(y_hat, y)
L.backward()

print("L =", L.item())          # ≈ 0.7126
print("dL/dW2 =", W2.grad)      # ≈ [[-0.3173, -0.3824]]
print("dL/db2 =", b2.grad)      # ≈ -0.5097
print("dL/dW1 =", W1.grad)      # ≈ [[-0.0599, -0.1198], [0.0573, 0.1146]]
print("dL/db1 =", b1.grad)      # ≈ [-0.0599, 0.0573]

If your numbers match to four decimals, you've correctly internalised every identity from 2.5–2.7.

7. What this exercise teaches

• Backprop is not magic. It is the chain rule applied carefully.
• Each $\delta$ has a clean shape: same as the corresponding $z$.
• Each $\partial L / \partial W$ is an outer product, same shape as the weight.
• Numerical mistakes show up immediately when the PyTorch reference disagrees with your hand-computed values.

8. Takeaways

• Pick a small net, one example, and run the full forward + backward by hand.
• Match every number to a reference implementation.
• After this exercise, the rest of Chapter 2 (vanishing/exploding, init, autodiff) is layered on a foundation you have actually computed.