1.3 — Gradient descent on the logistic-regression loss

1. Why this session exists

Session 1.2 gave us the loss and the pre-activation gradient $\partial L / \partial z = \hat y - y$. Now we walk the last step — from $\partial L / \partial z$ to $\partial L / \partial w$ — and write down the gradient-descent update. This is the simplest end-to-end training loop in the course: it has all the moving pieces of SGD with none of the depth.[1]

2. Setup

A training set $\{(x_i, y_i)\}_{i=1}^m$ with $x_i \in \mathbb{R}^n$ and $y_i \in \{0, 1\}$. Parameters $w \in \mathbb{R}^n$, bias $b \in \mathbb{R}$. Per-example pre-activation, prediction, loss:

Average BCE loss:

3. The gradients

From 1.2, $\partial \ell_i / \partial z_i = \hat y_i - y_i$. The chain rule extends this through the linear layer.

Gradient w.r.t. $w$. Since $\partial z_i / \partial w = x_i^\top$ (a $1 \times n$ row, numerator convention),

Transposing to a column gradient and averaging over the mini-batch,

Gradient w.r.t. $b$. $\partial z_i / \partial b = 1$, so

The residual $(\hat y_i - y_i)$ appears in both gradients the way it did for linear regression in 1.1 (with $\hat y_i$ being $w^\top x_i + b$ there). The pattern is general: gradient = average over examples of (residual × input).

4. The update rule

Vanilla (batch) gradient descent uses the full-batch gradient:

with learning rate $\alpha > 0$. Expanding,

This is the canonical update written out symbol by symbol. Three variants depending on how many examples we average over per step:

• Batch GD. All $m$ examples per step. Smooth trajectory, expensive when $m$ is large.
• Stochastic GD (SGD). One example $i_k$ per step: $w \leftarrow w - \alpha (\hat y_{i_k} - y_{i_k}) x_{i_k}$. Noisy but cheap.
• Mini-batch SGD. Average over a mini-batch of size $B$ (typically 32–1024 for tabular data, larger for vision/NLP). Standard practice.

5. Two pathologies and how to fix them

Separable data. If the classes are perfectly linearly separable, the likelihood can be made arbitrarily close to $1$ by scaling $w$ outwards: $\hat y = \sigma(\alpha w^\top x) \to \{0, 1\}$ as $\alpha \to \infty$ on either side of the hyperplane, and the log-likelihood approaches its supremum at $0$. The MLE doesn't exist — $w$ diverges. Fix: L2 regularisation $L \to L + \tfrac{\lambda}{2} \lVert w \rVert^2$, which makes the problem strictly convex with a unique finite minimiser.

Imbalanced classes. If one class dominates (say 99% negatives), the unregularised loss is dominated by the negative class and the model collapses to predicting $\hat y \approx 0$ everywhere. Fix: class weights in the loss, oversampling / undersampling the data, or focal-style losses (Chapter 7).

6. Where this is going

The update rule

is the single-sigmoid-unit version of the gradient descent rule that, in Chapter 2, will be composed across many hidden layers. The only new ingredient there is the chain rule from 0.3 walked backwards through additional matrix multiplications.

7. Takeaways

• BCE gradient: $\nabla_w L = \tfrac{1}{m} \sum_i (\hat y_i - y_i) x_i$ and $\partial L / \partial b = \tfrac{1}{m} \sum_i (\hat y_i - y_i)$.
• Pattern: residual × input, averaged.
• BCE on logistic regression is convex; SGD finds the global minimum (use L2 regularisation if data is separable).