3.4 — AdaGrad and RMSProp: per-parameter adaptive learning rates

1. Why this session exists

Momentum gives every parameter the same learning rate $\alpha$. That is a problem when some parameters see large gradients all the time (embeddings of frequent tokens, biases in the active part of a ReLU) and others see tiny ones (embeddings of rare tokens, weights in a saturated region). One global $\alpha$ is too aggressive for the noisy parameters and too cautious for the quiet ones.

The fix is per-parameter adaptive learning rates: divide each coordinate's step by a measure of how big its gradients have historically been. AdaGrad does this with a cumulative sum; RMSProp replaces the sum with an EMA. Adam will combine RMSProp with momentum in 3.5.

2. AdaGrad

Duchi et al. (2011)[1]. For each parameter coordinate $i$, keep a running sum of squared gradients

and use it to scale the step element-wise:

The tiny constant $\varepsilon \sim 10^{-8}$ prevents division by zero at $t = 0$. Read coordinate by coordinate: parameters whose gradients have been large get a smaller effective LR; parameters whose gradients have been small keep a large effective LR. The total "amount of learning" per parameter is roughly equalised.

Intuition by dimension analysis: $g_t$ has the same units as $\theta$ per unit time. $\sqrt{G_t}$ also has units of $\theta$. The ratio $g_t / \sqrt{G_t}$ is dimensionless, and $\alpha$ supplies the units of $\theta$. The update is unit-aware in a way plain SGD is not.

3. AdaGrad's failure mode

The sum $G_t^{(i)}$ is monotonically non-decreasing: every step it picks up another $(g_t^{(i)})^2 \ge 0$. So the effective learning rate $\alpha / \sqrt{G_t}$ decays as $\Theta(1/\sqrt{t})$ in the worst case and never recovers.

For convex problems this is provably optimal (AdaGrad attains the optimal online regret bound for adaptive methods, see Duchi et al.). For deep nets, where you might need thousands of epochs and the optimal step size changes through training, AdaGrad shuts itself off before reaching a good solution. By the end of training, $\sqrt{G_t}$ is so large that no step makes meaningful progress, even when the loss landscape is begging for more movement.

This is the bug. The fix is to forget the distant past.[4, 5, 6]

4. RMSProp

Hinton (Coursera, 2012[2] — never formally published; the canonical reference is the lecture slides). Replace the cumulative sum with an exponential moving average:

This is the EMA from §3.2 applied element-wise to the squared gradient, then used as a per-parameter scale. Typical $\beta_2 = 0.99$ (Hinton's slides) or $\beta_2 = 0.999$ (the value Adam will inherit). The effective averaging window is $\sim 1/(1-\beta_2) \sim 100$ to $1000$ steps.

Because $v_t$ is an EMA rather than a sum, the effective LR no longer decays to zero — it tracks the current variance of the gradient. If a parameter starts seeing smaller gradients, $v_t$ adapts down, and the effective LR for that parameter goes back up. AdaGrad's monotonic decay is gone.

5. The geometric reading: a diagonal preconditioner

Both AdaGrad and RMSProp are instances of diagonal preconditioning. Let $D_t = \operatorname{diag}(d_t^{(1)}, \dots, d_t^{(p)})$ be a positive diagonal matrix. A preconditioned gradient step is

For AdaGrad, $d_t^{(i)} = \sqrt{G_t^{(i)} + \varepsilon}$. For RMSProp, $d_t^{(i)} = \sqrt{v_t^{(i)} + \varepsilon}$.

Choosing $D_t$ correctly can dramatically reduce the effective condition number $\kappa(D_t^{-1} H)$ — where $H$ is the Hessian — and therefore speed up convergence (compare with the $\sqrt{\kappa}$ analysis of momentum in 3.3). A "perfect" $D_t = \sqrt{\operatorname{diag}(H)}$ would diagonalise the per-coordinate scaling; what RMSProp uses is a cheap proxy: $\sqrt{\operatorname{diag}(\mathbb{E}[g g^\top])}$, the diagonal of the gradient covariance. For convex quadratics with diagonal $H$, this proxy is right (because the gradient covariance is proportional to $H$); for deep nets it is approximate but useful.

This is also the answer to the common question "why $\sqrt{v}$ and not $v$?" — preconditioning a gradient step needs a length scale ($d_t \propto$ standard deviation of gradient, not variance), and $\sqrt{v_t}$ supplies it.[3]

6. Comparison to momentum

Momentum and RMSProp address different problems:

• Momentum smooths over time, in the same direction for all coordinates. It accelerates along directions of consistent sign.
• RMSProp rescales per coordinate, independent of sign. It equalises the magnitudes across coordinates.

They are complementary. Combine momentum's velocity $m_t$ (EMA of $g$) and RMSProp's scale $v_t$ (EMA of $g^2$), use $m_t / \sqrt{v_t}$ as the update direction, and you get Adam — next session.

7. Practical defaults

For RMSProp:

• $\alpha = 10^{-3}$.
• $\beta_2 = 0.99$ (Hinton's value; PyTorch default alpha=0.99).
• $\varepsilon = 10^{-8}$.
• No bias correction (you can apply the same $1/(1 - \beta_2^t)$ trick from 3.2; some implementations do, most don't).

RMSProp was the workhorse adaptive optimiser from 2012 until Adam's paper in 2014. It is still used in RL (where Adam's momentum can fight policy updates) and in some image-generation pipelines.

8. Takeaways

• AdaGrad: cumulative $G_t = \sum (g_t)^2$; per-parameter LR $\alpha / \sqrt{G_t}$. Provably optimal for convex problems; stalls in deep learning because $G_t$ never decreases.
• RMSProp: EMA $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$; same scaling but adaptive over a sliding window.
• Both are diagonal preconditioners — they reshape the gradient by $1/\sqrt{\text{coordinate variance}}$.
• The $\sqrt{}$ is not optional: it makes the units of the update match the units of $\theta$.
• RMSProp + momentum → Adam (3.5).