0.6 — Entropy and KL divergence

1. Shannon information of a single outcome

If an event with probability $p$ occurs, the information it carries is

The minus sign makes $I$ non-negative (since $p \le 1$, $\log p \le 0$). Three sanity checks:

• $I(1) = 0$: a certain outcome carries zero information.
• $I(p) \to \infty$ as $p \to 0$: an impossible-looking outcome that nonetheless occurs is maximally informative.
• $I(p_1 p_2) = I(p_1) + I(p_2)$: information from two independent events adds. This additivity is what fixes the form of $I$ uniquely up to a log base.

Base of the log sets the units: base 2 gives bits, base $e$ gives nats. In ML we use natural log throughout (gradient-friendly); the formulas are identical up to a constant factor.[1]

2. Entropy

Entropy is the expected information of a draw from $p$:

For continuous distributions, replace the sum with an integral; the quantity is then called differential entropy and has subtler properties (it can be negative, isn't invariant under change of variables) — but the formula is the same shape.

Properties:

• $H[p] \ge 0$ for discrete $p$ (each term $-p \log p$ is non-negative on $[0, 1]$, with the convention $0 \log 0 = 0$).
• $H[p]$ is maximised by the uniform distribution. Intuition: the uniform distribution is "maximally unpredictable", carrying the most information per draw.
• $H$ is concave in $p$ — a fact we won't need today but that explains why "averaging two distributions doesn't reduce uncertainty".

3. Cross-entropy

If $p$ is the true distribution and $q$ is the one you code with (a model), the cross-entropy is

Operationally: $H[p, q]$ is the expected number of bits (or nats) you spend encoding draws from $p$ using an optimal code for $q$. When $q = p$, you've made the right choice and $H[p, q] = H[p]$. When $q \neq p$, you waste bits — and the wasted bits are what KL divergence measures.

4. KL divergence

The Kullback–Leibler divergence of $p$ from $q$ is

It satisfies the identity

which makes the "wasted bits" reading precise. Worked out:

KL is not symmetric: $\mathrm{KL}(p\,\|\,q) \neq \mathrm{KL}(q\,\|\,p)$ in general. It is not a distance — it's a divergence. The first argument is "the truth"; the second is "your model".

5. KL is non-negative (Jensen's inequality)

We want to show $\mathrm{KL}(p\,\|\,q) \ge 0$, with equality iff $p = q$.

The key tool is Jensen's inequality for a concave function $\varphi$ and a random variable $Z$:

The function $\log$ is concave on $(0, \infty)$. So:

Therefore $\mathrm{KL}(p\,\|\,q) \ge 0$. Equality holds iff $q(x)/p(x)$ is constant under $p$ — which forces $q = p$ since both are probability distributions.

This is the fundamental inequality of probabilistic ML. Almost every "loss is bounded below by zero" or "evidence lower bound" argument in the field passes through it.

6. Information-theoretic identities you will use

A handful of identities that follow by direct expansion and that we'll invoke without re-deriving from here on:

• Mutual information. $I(X; Y) = \mathrm{KL}(p_{XY}\,\|\,p_X p_Y) = H[X] - H[X \mid Y]$. Measures how much knowing $Y$ reduces uncertainty about $X$. Non-negative; zero iff $X \perp Y$.
• Conditional entropy. $H[X \mid Y] = -\mathbb{E}_{(X,Y)}[\log p(X \mid Y)]$. Average uncertainty about $X$ remaining after $Y$ is observed.
• Chain rule of entropy. $H[X, Y] = H[X] + H[Y \mid X] = H[Y] + H[X \mid Y]$. Mirror of the product rule of probability.

7. Why this matters for what's next

Cross-entropy as a training loss (§0.7) is exactly $H[p_{\text{data}}, q_{\theta}]$, and its gradient with respect to $\theta$ doesn't see the data-only term $H[p_{\text{data}}]$ — making it operationally identical to negative log-likelihood. KL appears explicitly in variational autoencoders, in RLHF's KL penalty against a reference policy, and as the constraint in trust-region methods (TRPO, PPO). Mutual information is the lens used to discuss compression, the information-bottleneck theory of neural networks, and self-supervised contrastive losses. One vocabulary, used everywhere.