Chapter 0 · 18 min

0.1 — Vector spaces and inner products, the ML view

Goal: Re-derive ⟨x,y⟩ = xᵀy and convince yourself a linear layer is an affine map.

1. Where the data lives

In ML, the underlying object of any data point is a vector in a finite-dimensional real vector space $V = \mathbb{R}^d$ . A 28×28 grayscale image becomes a vector with $d = 784$ components; a sequence of $T$ tokens in a language model, once embedded, becomes a matrix in $\mathbb{R}^{T\times d_\text{model}}$ , but each row is a vector in $\mathbb{R}^{d_\text{model}}$ .

The choice of $\mathbb{R}^d$ is not innocent — it commits us to three things at once: additive structure ( $x + y \in V$ ), scalar multiplication ( $\lambda x \in V$ ), and the existence of a canonical inner product, which we'll now derive.

2. Inner product via basis expansions

Fix a basis $\{e_1, \dots, e_d\}$ of $V$ . Every vector has a unique expansion

x = \sum_{i=1}^{d} x_i \, e_i, \qquad x_i \in \mathbb{R}.

Suppose $V$ is equipped with a symmetric positive-definite bilinear form $\langle \cdot, \cdot \rangle : V \times V \to \mathbb{R}$ . By bilinearity, for $y = \sum_j y_j e_j$ ,

\langle x, y \rangle = \Big\langle \sum_i x_i e_i,\, \sum_j y_j e_j \Big\rangle = \sum_{i,j} x_i\, y_j\, \langle e_i, e_j \rangle.

When the basis is orthonormal, $\langle e_i, e_j \rangle = \delta_{ij}$ , and the double sum collapses:

\boxed{\;\langle x, y \rangle = \sum_i x_i y_i = x^\top y.\;}

This is the form that shows up in every ML derivation: inner product is row-times-column multiplication.

3. Induced norm and angle

The norm induced by the inner product is

\|x\| = \sqrt{\langle x, x \rangle} = \Big( \sum_i x_i^2 \Big)^{1/2}.

And the angle $\theta$ between $x$ and $y$ is defined implicitly by

\cos\theta = \frac{\langle x, y \rangle}{\|x\|\, \|y\|}.

The cosine similarity used in semantic search and embedding normalisation is this ratio. When vectors are pre-normalised to unit norm, $\cos\theta = x^\top y$ — that's it.

4. A linear layer is an affine map

A fully connected layer with weights $W \in \mathbb{R}^{m\times d}$ and bias $b \in \mathbb{R}^m$ implements the map

f : \mathbb{R}^d \to \mathbb{R}^m, \qquad f(x) = Wx + b.

This is the classical definition of an affine transformation: linear ( $x \mapsto Wx$ ) plus translation ( $+\,b$ ). The bias trick — appending a $1$ to the end of $x$ and stacking $b$ as the last column of $W$ — turns an affine map into a purely linear one in the augmented space $\mathbb{R}^{d+1}$ . This simplifies proofs, but in code we keep $b$ separate for efficiency (it broadcasts).

5. Where this is going

All the algebra to come — attention, softmax, layer normalisation — rests on these three facts: there is a basis, there is an inner product, and layers are affine. Internalise them; there's no shortcut.