4.B — Pooling: max and average (appendix)

1. Why this appendix exists

Pooling is the second pillar of a classical CNN — convolution provides equivariant feature extraction, pooling provides local invariance and downsampling. Pooling layers have no learnable parameters, but they shape the receptive-field calculation and they have a slightly subtle backward pass that is worth writing once.[1]

2. The pooling forward pass

Pooling slides a window of size $P_h \times P_w$ over the spatial dimensions of the input — without crossing channels — and outputs a single number per window.

Max-pool:

Average-pool:

Both operate independently per channel — there is no learned mixing between channels in a pooling layer. The stride $S$ is typically equal to the window size so that pooling windows don't overlap (the "standard" $2 \times 2$ max-pool with stride 2).

Output size follows the same formula as in Appendix 4.A:

with $P = 0$ for "valid" pooling, the usual choice.

3. Local translation invariance

Pooling provides local translation invariance — the output is unchanged by small shifts of the input within a pool window — at the cost of throwing away spatial resolution.

To see this: if you shift the input by less than the pool size, the set of values inside each window is (usually) unchanged. Max-pool depends only on the set, so its output is unchanged. Average-pool changes slightly because the averaged values are weighted equally, but only by what enters and leaves the window at the boundary. The shift the layer can absorb is bounded by $P_h$ or $P_w$.

A stack of pooling layers compounds: $L$ pool layers of size 2 with stride 2 are invariant to shifts up to $\sim 2^L$ pixels. This is the behaviour required of an image classifier, where small object translations should not change the class label.

4. The backward pass

Both pooling layers are non-linear maps with no learnable parameters, but they still need a backward pass to propagate gradients to the input.

4.1 Max-pool backward

Max is sub-differentiable, but at any input where the maximiser is unique, the gradient routes the upstream signal entirely to that single position:

In code: cache the argmax during the forward pass and use it as a routing table on the backward.

If the maximiser is not unique, any selection works (the layer is non-differentiable on a measure-zero set; we already lived with this for ReLU in 0.9). Frameworks pick one deterministically and move on.

4.2 Average-pool backward

Average-pool is linear, so its backward is just the linearisation:

for every $(k, l)$ in the pool window of $(i, j)$ (and zero outside any window if windows are non-overlapping).

Average-pool's gradient is spread uniformly over the pool window; max-pool's gradient is concentrated on the maximiser. This is the structural difference between the two layers and is why max-pool is occasionally said to "harden" features while average-pool "softens" them.

5. Modern practice

In LLM-era architectures, pooling has largely fallen out of favour. Vision Transformers replace pooling with patch tokenisation plus attention. ResNet-style CNNs still use one or two pool or strided-conv layers near the input. In NLP, pooling never had much currency to begin with — the only common operation is "pool over the sequence dimension" to summarise a variable-length sequence into a single vector, which is what attention's [CLS] token or final-token readout does instead.