2.10 — Weight initialisation: Xavier/Glorot and He

1. Why this session exists

Section 2.9 left an open question: even if the activation $g$ has well-behaved $g'$, the matrix factors $(W^{(\ell+1)})^\top$ in the backprop chain can themselves rescale the gradient. To keep activations and gradients well-scaled across depth, we initialise weights with a carefully chosen variance. The two recipes are Xavier/Glorot (linear or tanh) and He (ReLU).

This is one of those moments where deep learning's empirical fix turns out to come from a one-line calculation.

2. Setup: variance propagation through a linear layer

Consider one linear layer

with $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, $a \in \mathbb{R}^{d_{\text{in}}}$, $b = 0$ (for now). Assume entries of $W$ are i.i.d. with mean $0$ and variance $\sigma_W^2$, independent of $a$. Then

Compute $\operatorname{Var}(z_i)$. With $\mathbb{E}[W_{i, j}] = 0$ and independence,

(With $a_j$ centred and i.i.d. across $j$, $\mathbb{E}[a_j^2] = \operatorname{Var}(a)$.)

To preserve variance through the layer, $\operatorname{Var}(z) = \operatorname{Var}(a)$, we need

This is the variance-preservation principle for forward propagation.

3. Xavier / Glorot initialisation (Glorot & Bengio, 2010)[1]

Glorot's full analysis does the same calculation in the backward pass and asks for both forward and backward variances to be preserved. Backward through the linear layer (from 2.6),

To preserve $\operatorname{Var}(\delta)$ we need $\sigma_W^2 = 1 / d_{\text{out}}$. Forward wants $1 / d_{\text{in}}$; backward wants $1 / d_{\text{out}}$. Compromise: take the harmonic mean,

In practice we sample $W_{i, j} \sim \mathcal{N}(0, \sigma_W^2)$ or, equivalently variance-wise, $W_{i, j} \sim \mathcal{U}(-r, r)$ with $r = \sqrt{6 / (d_{\text{in}} + d_{\text{out}})}$.

This was the right choice when the dominant activation was tanh, which has $\tanh'(0) = 1$ — so the activation doesn't change the variance budget. With sigmoid hidden layers, Xavier was already a marked improvement on the previous "random scale, hope" approach.

4. He initialisation (He et al., 2015)[2]

For ReLU activations the calculation needs one more step. Recall (2.3, 2.9) that $\operatorname{ReLU}'(z) \in \{0, 1\}$ and that for a symmetrically distributed $z$, $\operatorname{ReLU}(z) = 0$ half the time. Plugging this into the variance propagation:

assuming a centred symmetric $z$. So passing through a ReLU halves the variance. To compensate, the next layer's weight matrix should double the variance budget:

This is the He (or Kaiming) initialisation — the default for ReLU networks today (nn.init.kaiming_normal_ in PyTorch).

Why this matters concretely. Without the $2 / d_{\text{in}}$ correction, ReLU networks see their activation variance halve at every layer; after $L = 20$ layers, the typical activation magnitude has shrunk by $2^{-10} \approx 10^{-3}$, and the network is effectively dead. He init keeps the activation variance constant across depth.

5. Generalisations

Two natural extensions.

Generic activation $g$ with derivative-magnitude $\eta := \sqrt{\mathbb{E}[(g'(Z))^2]}$ at the initialisation scale. The compensating variance is

Plugging $\eta^2 = 1/2$ (ReLU half-on) gives $2 / d_{\text{in}}$, recovering He. Plugging $\eta = 1$ (tanh near zero) gives $1 / d_{\text{in}}$, recovering Xavier-forward.

Layer-wise scale rescue. Modern alternatives like LSUV (Mishkin & Matas, 2016) iteratively rescale each layer's weights so that the actual variance of $z^{(\ell)}$ at initialisation matches a target value, measured on a real batch. More accurate than the closed-form rule, more annoying to implement.

6. Biases

Biases are usually initialised to zero. There is no variance to preserve — biases shift means, not variances. Exception: the bias of the output logit for a binary classifier on imbalanced data is sometimes initialised to $\operatorname{logit}(\mathbb{P}(y = 1))$ so that the network's prior prediction at step 0 matches the class frequency.

7. A common pitfall

Many older codebases (and CV pipelines from the early 2010s) use the forward-only Xavier variance $\operatorname{Var}(w^{(\ell)}) = 1 / d_{\ell - 1}$ across the board — tanh or ReLU, indifferently. For tanh-activated networks this is fine. For ReLU networks it under-scales the weights by a factor of $2$ in variance, and the activation variance halves at every layer (§4). The fix is one line: switch the scaling factor from $1/d_{\text{in}}$ to $2/d_{\text{in}}$ (or call nn.init.kaiming_normal_ in PyTorch) and trainability improves immediately. A small but real example of how the theory keeps catching up to the practice.

8. Takeaways

• Forward-variance preservation: $\sigma_W^2 = 1 / d_{\text{in}}$.
• Backward-variance preservation: $\sigma_W^2 = 1 / d_{\text{out}}$.
• Xavier (compromise): $\sigma_W^2 = 2 / (d_{\text{in}} + d_{\text{out}})$. Use with linear or tanh.
• He (ReLU correction): $\sigma_W^2 = 2 / d_{\text{in}}$. Use with ReLU.
• Biases initialise to zero unless you have prior class-imbalance information.