4.A — Padding, stride, and output-size formulas (appendix)

1. Why this appendix exists

The forward formula in 4.1 §4 assumed no padding and stride 1. Real conv layers expose two extra knobs — padding $P$ and stride $S$ — that control output size and how aggressively the layer downsamples. The formulas are simple but produce off-by-one bugs when you misremember which way to round.[1]

2. Stride

A convolution with stride $S$ shifts the kernel by $S$ positions between consecutive outputs instead of by 1. The forward formula becomes

For $S = 1$ this reduces to the standard cross-correlation. For $S = 2$ the output is roughly half the input size in each spatial dimension — i.e. it downsamples by a factor of 2.

3. Padding

Padding wraps the input with $P$ extra rows and columns of zeros on each side before convolving. This serves two purposes: it lets the kernel "fit" at boundary pixels (otherwise the output is strictly smaller), and it gives precise control over output size.

Three padding choices appear constantly:

• Valid padding ($P = 0$). No padding. Output is strictly smaller than input.
• Same padding ($P$ chosen so output size $=$ input size). For stride 1, this means $P = (K - 1)/2$, which is only an integer when $K$ is odd — one of the two reasons odd kernel sizes ($3, 5, 7$) are standard.
• Full padding ($P = K - 1$). The kernel touches every input position at least once and the output is larger than the input. Rarely used in CNN forward passes; appears naturally in the backward pass of conv (Appendix 4.C).

4. The general output-size formula

With padding $P$ on each side and stride $S$, the output spatial size is:

Derivation. After padding, the effective input length is $H_\text{in} + 2P$. The kernel of size $K$ has $H_\text{in} + 2P - K + 1$ valid starting positions on a unit-stride grid. With stride $S$, we take every $S$-th starting position, giving $\lfloor (H_\text{in} + 2P - K) / S \rfloor + 1$ outputs (the $+1$ counts the position at offset 0).

The same formula applies to width. For 1D convolutions, $H_\text{in}$ is the sequence length.

4.1 Sanity checks

• Valid, stride 1. $P = 0, S = 1$: $H_\text{out} = H_\text{in} - K + 1$. Matches 4.1 §4.
• Same, stride 1, odd $K$. $P = (K - 1)/2, S = 1$: $H_\text{out} = (H_\text{in} + (K - 1) - K)/1 + 1 = H_\text{in}$. ✓
• Stride 2, $K = 3, P = 1$. $H_\text{out} = \lfloor (H_\text{in} + 2 - 3)/2 \rfloor + 1 = \lfloor (H_\text{in} - 1)/2 \rfloor + 1$. For $H_\text{in} = 32$: $\lfloor 31/2 \rfloor + 1 = 15 + 1 = 16$. Halved, as expected.

5. Dilation

A kernel with dilation $d$ inserts $d - 1$ implicit zeros between adjacent kernel weights. The kernel covers a span of $(K - 1) d + 1$ input positions while using only $K$ learnable weights.

The output-size formula generalises to:

Set $d = 1$ to recover §4. Dilated stacks (WaveNet, dilated TCNs) use this to grow receptive field exponentially with depth without any extra parameters or strides — see 4.3 §2 and the Aside in 4.2.

6. Practical defaults

For Transformer-era engineers who pass through a conv layer occasionally (vision encoders, audio frontends), the default is $K = 3$, stride 1, same padding, dilation 1. Downsampling uses stride 2 with same padding (or, in vision backbones, a separate pooling layer). Almost every "off the shelf" CNN block can be reconstructed from this with at most two extra knobs.