3.A — L2 weight decay as a Gaussian prior (MAP) [appendix]

Optional appendix. This session gives the Bayesian reading of weight decay (already used pragmatically in 3.6 AdamW). It does not block Chapters 4–12 — skip if you only want the operational picture.

1. Why this session exists

Weight decay is by far the most common regulariser in deep learning. We've already seen the mechanics (3.6: a $(1 - \alpha \lambda)\,\theta$ shrinkage each step). What we haven't done is justify why shrinking weights toward zero should help.

The standard justification is Bayesian: weight decay is the MAP estimate of $\theta$ under a Gaussian prior on the weights. The penalty strength $\lambda$ is one-over-prior-variance. Once you see the derivation, the choice of L2 (rather than L1 or L4 or anything else) stops being arbitrary — it corresponds to the Gaussian assumption, which is the "most ignorant" symmetric prior in the maximum-entropy sense.

2. Recap: MLE and the loss

From 0.7: the maximum-likelihood estimator (MLE) of $\theta$ given data $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ is

For Gaussian regression $p(y \mid x, \theta) = \mathcal{N}(f_\theta(x), \sigma^2)$, the right-hand side is (up to a constant) the squared-error loss. For categorical classification with softmax outputs, it is the cross-entropy. The training loss $L(\theta)$ is — by design — proportional to the negative log-likelihood.

3. Adding a prior: MAP

Now treat $\theta$ as a random variable with a prior $p(\theta)$. By Bayes' rule (0.5),

The maximum-a-posteriori (MAP) estimate is

This is the MLE objective with an extra term $-\log p(\theta)$ — the negative log-prior, acting as a regulariser. The shape of this term encodes our beliefs about $\theta$ before seeing the data.[1]

4. The Gaussian prior

Pick the most basic symmetric prior: each parameter $\theta_i$ is independent and Gaussian with mean $0$ and variance $\tau^2$. Then

so

Substitute into the MAP objective:

Comparing to the L2-penalised loss $L(\theta) + \tfrac{\lambda}{2} \lVert \theta \rVert^2$, we read off

The penalty strength is the reciprocal of the prior variance. Large $\lambda$ ↔ tight prior (we believe weights should be small). Small $\lambda$ ↔ broad prior (we are agnostic). $\lambda = 0$ ↔ infinite prior variance ↔ pure MLE.

This is the precise statement of "L2 regularisation = Gaussian prior on weights".

5. Why Gaussian, why on weights?

Three reasons to like this choice.

(a) Maximum entropy. Among all distributions on $\mathbb{R}$ with a given variance, the Gaussian is the one with maximum entropy (0.6). It is the least-committal symmetric prior compatible with "weights should be around zero, with some characteristic scale".

(b) Smoothness of $f$. For a linear model $f(x) = \theta^\top x$, the gradient w.r.t. input is $\nabla_x f = \theta$, so $\lVert \nabla_x f \rVert = \lVert \theta \rVert$. Small $\lVert \theta \rVert$ ↔ small Lipschitz constant ↔ output changes little under small input perturbations. The L2 penalty encourages a smoother predictor, which generalises better when the true target function is smooth. The same intuition extends to deep nets, where each layer's weight matrix contributes to the overall Lipschitz constant.

(c) Numerical stability. Big weights amplify activations, which amplify gradients, which can explode. Keeping $\lVert \theta \rVert$ bounded keeps the forward/backward passes in a sane numeric range — relevant for fp16 training of LLMs.

What about biases? Under the same Gaussian prior, biases would also be shrunk. In practice we do not shrink biases. The justification: biases set the operating point of each neuron's activation (e.g., they determine where a ReLU starts firing); there is no statistical reason to believe operating points should cluster around zero. The bias on the output layer in particular calibrates the mean of the prediction and is left free. This matches the AdamW convention from 3.6 of excluding biases from weight decay.

6. What about L1?

The same derivation with a Laplace prior $p(\theta_i) \propto \exp(-|\theta_i| / b)$ yields

So L1 regularisation = Laplace prior on weights. The minimiser of $L + \lambda \lVert \theta \rVert_1$ tends to set many $\theta_i$ exactly to zero (sparsity), because the Laplace prior is sharply peaked at zero. This is the basis of LASSO regression.

In deep learning L1 is rare for hidden-layer weights — sparsity in weights doesn't translate to sparsity in activations, and dense weights are typically more accurate. L1 reappears in specific contexts: sparse coding, structured pruning, mixture-of-experts gating losses.

7. The Hessian view (deep nets, briefly)

The MAP objective is

Its Hessian is

The L2 term shifts every eigenvalue of the Hessian up by $\lambda$. This regularises the conditioning: directions in parameter space that were nearly flat under $L$ alone (eigenvalues near zero — many in deep nets, the "degenerate" directions) become $\lambda$-curved under $\tilde L$. The optimiser sees a strictly convex bowl in those directions and converges instead of wandering. This is the same trick as Levenberg–Marquardt or damped Newton; weight decay is essentially a free Tikhonov regulariser on the parameter geometry.

8. Takeaways

• The L2-penalised loss is the MAP estimate under an independent Gaussian prior on each parameter, with mean zero and variance $\tau^2$.
• The penalty coefficient is $\lambda = 1 / \tau^2$. Large $\lambda$ ↔ tight prior.
• L1 corresponds to a Laplace prior (sparsity-inducing). Used in sparse models, rare for hidden-layer weights of deep nets.
• Biases are typically excluded — no reason to believe operating points cluster at zero.
• Hessian view: L2 shifts every eigenvalue of $\nabla^2 L$ up by $\lambda$, regularising flat directions of the loss landscape.