0.7 — Cross-entropy as a loss, and why minimising it is MLE

1. The starting point: i.i.d. data and a parametric model

We have a training set $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{m}$ assumed to be i.i.d. draws from some unknown joint $p_{\text{data}}(x, y)$. We specify a parametric conditional model $q_\theta(y \mid x)$ and want to fit $\theta$ to the data.

(I write $q$ instead of $p$ for the model to keep the truth/model distinction explicit — same convention as §0.6.)[1]

2. The likelihood

Under the i.i.d. assumption, the joint probability the model assigns to the labels given the inputs factorises:

This is the conditional likelihood of the labels under the model. As a function of $\theta$ with the data fixed, it is denoted $\mathcal{L}(\theta)$.

3. Maximum likelihood estimation (MLE)

Maximum likelihood picks $\theta$ that maximises $\mathcal{L}(\theta)$. Products are awful to differentiate, so take the log (which is monotone and so doesn't change the argmax):

Flip the sign to turn maximisation into minimisation (the convention in optimisation):

This is the negative log-likelihood (NLL). Divide by $m$ to get the per-sample version (gradients are then unitless in batch size):

4. The empirical distribution

Define the empirical distribution as the discrete measure that puts mass $1/m$ on each training point:

Expectation under $\hat p$ is "average over the training set", which we already noted in §0.4:

5. NLL is cross-entropy

Apply this to $f(x, y) = -\log q_\theta(y \mid x)$:

That last line is the cross-entropy between $\hat p$ and the conditional model $q_\theta$, written with the joint factored as $\hat p(x, y) = \hat p(x)\,\hat p(y \mid x)$:

i.e. the input-averaged cross-entropy between the empirical conditional $\hat p(y \mid x)$ and the model conditional $q_\theta(y \mid x)$.

Putting it together:

MLE on a discriminative model is exactly cross-entropy minimisation on the empirical distribution. The two names refer to the same gradient.

6. Why the model's "missing" entropy term doesn't matter for gradients

Recall from §0.6 that $H[\hat p, q_\theta] = H[\hat p] + \mathrm{KL}(\hat p\,\|\,q_\theta)$. So minimising cross-entropy over $\theta$ is identical to minimising $\mathrm{KL}(\hat p\,\|\,q_\theta)$, since $H[\hat p]$ doesn't depend on $\theta$ and drops out of the gradient. Three names for the same optimisation:

• MLE (statistics).
• Cross-entropy minimisation (information theory).
• Forward KL minimisation against the empirical distribution (Bayesian ML).

7. The two cases you'll see daily

Binary classification with $y \in \{0, 1\}$ and $q_\theta(y = 1 \mid x) = \hat y$ for some predicted probability:

These two cases unify into one expression — the binary cross-entropy loss:

Multi-class classification with $y \in \{1, \dots, K\}$ and $q_\theta(y = k \mid x) = \hat y_k$ for predicted class probabilities summing to 1 (e.g. coming from a softmax — see Ch. 1):

The last form ($-\log \hat y_y$) is what you read in code: take the predicted probability of the true class, take its log, negate. That's PyTorch's F.cross_entropy (modulo the fact that PyTorch's cross_entropy expects pre-softmax logits and applies the softmax internally for numerical stability — more on that in §0.10).

8. What we've earned

The same loss object reads three ways depending on the lens. Picking the right lens unstrucks problems:

• Statistical lens (MLE): what data-generating assumption does my loss implicitly make? E.g. mean-squared error ↔ Gaussian noise (Ch. 1).
• Information-theoretic lens (cross-entropy / KL): how far is my model from the empirical distribution? Variational and distillation methods sit here.
• Operational lens (NLL): what does the gradient look like and how do I implement it? See §0.10 for the stability tricks that make this practical.

For every loss in the rest of the course, ask which of the three views makes the calculation easy. Use that one.