AI / ML Theory
references

References and bibliography

Goal: Curated bibliography for every chapter — primary papers, textbooks, lecture series, and online resources.

This page collects every primary source cited in the course. It grows as chapters are written. Sources are grouped per chapter; within each group, items are listed in order of first appearance. Arxiv numbers and stable URLs are provided where possible.

Chapter 0 — Math foundations

Textbooks.

  • Boyd, S., & Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press. — Used in 0.8 (convex functions, Jensen, first/second-order conditions). stanford.edu/~boyd/cvxbook
  • Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley. — Used in 0.6 (entropy, KL divergence) and 0.7 (cross-entropy as MLE).
  • Magnus, J. R., & Neudecker, H. (2019). Matrix Differential Calculus with Applications in Statistics and Econometrics (3rd ed.). Wiley. — Standard reference for the numerator-convention matrix calculus of 0.2.
  • Higham, N. J. (2002). Accuracy and Stability of Numerical Algorithms (2nd ed.). SIAM. — Numerical-stability material in 0.10 (overflow, underflow, log-sum-exp).

Historical notes / theorems cited.

  • Rademacher, H. (1919). "Über partielle und totale Differenzierbarkeit von Funktionen mehrerer Variabeln und über die Transformation der Doppelintegrale". Math. Annalen, 79, 340–359. — The almost-everywhere differentiability theorem stated in 0.9.
  • Jensen, J. L. W. V. (1906). "Sur les fonctions convexes et les inégalités entre les valeurs moyennes". Acta Math., 30, 175–193. — Jensen's inequality (0.8).

Online resources.

  • bfloat16 specification. Wang & Kanwar (2019), "BFloat16: The secret to high performance on Cloud TPUs". cloud.google.com — Used in 0.10.

Chapter 1 — Linear models and the perceptron

Textbooks.

  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. — Linear regression (1.1), logistic regression (1.2–1.3), softmax (1.7), MAP / Gaussian priors.
  • Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. — Logistic regression, linear methods.

Primary papers.

  • Rosenblatt, F. (1958). "The perceptron: A probabilistic model for information storage and organization in the brain". Psychological Review, 65, 386–408. — Perceptron model (1.5).
  • Novikoff, A. B. J. (1962). "On convergence proofs on perceptrons". Symposium on the Mathematical Theory of Automata, 12, 615–622. — Mistake bound for the perceptron (1.5).
  • Minsky, M., & Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press. — The XOR impossibility result (1.6).

Chapter 2 — MLPs and backpropagation

Textbooks.

  • Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. — General reference for Chapter 2; specifically §6 (deep feed-forward networks), §8 (optimization), §6.4 (UAT). deeplearningbook.org
  • Griewank, A., & Walther, A. (2008). Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (2nd ed.). SIAM. — Canonical AD reference (2.11).

Primary papers.

  • Cybenko, G. (1989). "Approximation by superpositions of a sigmoidal function". Mathematics of Control, Signals, and Systems, 2(4), 303–314. — UAT for sigmoidal activations (2.2).
  • Hornik, K. (1991). "Approximation capabilities of multilayer feedforward networks". Neural Networks, 4(2), 251–257. — UAT for general non-polynomial activations (2.2).
  • Telgarsky, M. (2016). "Benefits of depth in neural networks". COLT. arXiv:1602.04485 — Depth-vs-width separation (2.2).
  • Eldan, R., & Shamir, O. (2016). "The power of depth for feedforward neural networks". COLT. arXiv:1512.03965 — Depth-2 vs depth-3 separation (2.2).
  • Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors". Nature, 323(6088), 533–536. — Backpropagation (2.5–2.7).
  • Baur, W., & Strassen, V. (1983). "The complexity of partial derivatives". Theoretical Computer Science, 22(3), 317–330. — Cost-of-gradient = cost-of-function theorem (2.11).
  • Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU Munich. — First vanishing-gradient analysis in RNNs (2.9).
  • Bengio, Y., Simard, P., & Frasconi, P. (1994). "Learning long-term dependencies with gradient descent is difficult". IEEE Transactions on Neural Networks, 5(2), 157–166. — Formal vanishing-gradient analysis (2.9).
  • Glorot, X., & Bengio, Y. (2010). "Understanding the difficulty of training deep feedforward neural networks". AISTATS. — Xavier/Glorot initialisation (2.10).
  • He, K., Zhang, X., Ren, S., & Sun, J. (2015). "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification". ICCV. arXiv:1502.01852 — He / Kaiming initialisation (2.10).
  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). "ImageNet classification with deep convolutional neural networks". NeurIPS. — AlexNet; popularised ReLU at scale (2.3).
  • Hendrycks, D., & Gimpel, K. (2016). "Gaussian Error Linear Units (GELUs)". arXiv:1606.08415 — GELU activation (2.3).
  • Ramachandran, P., Zoph, B., & Le, Q. V. (2017). "Searching for activation functions". arXiv:1710.05941 — Swish / SiLU (2.3).
  • Shazeer, N. (2020). "GLU variants improve Transformer". arXiv:2002.05202 — SwiGLU, GeGLU (2.3, will reappear in Cap 8).

Implementation references.

  • Karpathy, A. "Neural Networks: Zero to Hero" lecture series, especially micrograd (Lecture 1). karpathy.ai/zero-to-hero — Used as resource in 2.12.

Chapter 3 — Optimization and regularization

Primary papers — optimisers.

  • Polyak, B. T. (1964). "Some methods of speeding up the convergence of iteration methods". USSR Comp. Math., 4(5), 1–17. — Heavy-ball momentum (3.3).
  • Nesterov, Y. (1983). "A method for solving the convex programming problem with convergence rate O(1/k2)O(1/k^2)". Soviet Math. Dokl., 27, 372–376. — Nesterov accelerated gradient (3.3).
  • Duchi, J., Hazan, E., & Singer, Y. (2011). "Adaptive subgradient methods for online learning and stochastic optimization". JMLR, 12, 2121–2159. — AdaGrad (3.4).
  • Hinton, G. (2012). "Lecture 6e: rmsprop — divide the gradient by a running average of its recent magnitude". Coursera: Neural Networks for Machine Learning. — RMSProp; never published as a paper, the slides are the canonical reference (3.4).
  • Kingma, D. P., & Ba, J. (2014). "Adam: A method for stochastic optimization". arXiv:1412.6980 — Adam optimiser (3.5).
  • Loshchilov, I., & Hutter, F. (2019). "Decoupled weight decay regularization". ICLR. arXiv:1711.05101 — AdamW (3.6).

Primary papers — LR schedules and gradient clipping.

  • Loshchilov, I., & Hutter, F. (2017). "SGDR: Stochastic gradient descent with warm restarts". ICLR. arXiv:1608.03983 — Cosine schedule (3.7).
  • Pascanu, R., Mikolov, T., & Bengio, Y. (2013). "On the difficulty of training recurrent neural networks". ICML. arXiv:1211.5063 — Gradient clipping by global norm (3.7).
  • Hu, S., Tu, Y., Han, X., et al. (2024). "MiniCPM: Unveiling the potential of small language models with scalable training strategies". arXiv:2404.06395 — WSD schedule (3.7).
  • Liu, L., Jiang, H., He, P., et al. (2020). "On the variance of the adaptive learning rate and beyond". ICLR. arXiv:1908.03265 — Variance analysis of warmup (3.7).
  • Brown, T., Mann, B., Ryder, N., et al. (2020). "Language models are few-shot learners". NeurIPS. arXiv:2005.14165 — GPT-3 training schedule; referenced in 3.7.
  • Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). "Training compute-optimal large language models". arXiv:2203.15556 — Chinchilla schedule recipe (3.7).

Primary papers — normalisation.

  • Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). "Layer normalization". arXiv:1607.06450 — LayerNorm (3.8).
  • Zhang, B., & Sennrich, R. (2019). "Root mean square layer normalization". NeurIPS. arXiv:1910.07467 — RMSNorm (3.8).
  • Ioffe, S., & Szegedy, C. (2015). "Batch normalization: Accelerating deep network training by reducing internal covariate shift". ICML. arXiv:1502.03167 — BatchNorm (Appendix 3.C).
  • Wu, Y., & He, K. (2018). "Group normalization". ECCV. arXiv:1803.08494 — GroupNorm (mentioned in 3.8).

Primary papers — regularisation (Cap 3 appendices).

  • Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). "Dropout: A simple way to prevent neural networks from overfitting". JMLR, 15, 1929–1958. — Dropout (Appendix 3.B).
  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning, §3.4 and §5.5. — L2 as Gaussian prior / MAP (Appendix 3.A).

Modern alternative optimisers (mentioned in asides).

  • Chen, X., Liang, C., Huang, D., et al. (2023). "Symbolic discovery of optimization algorithms" (Lion). arXiv:2302.06675 — Lion (3.6 aside).
  • Liu, H., Li, Z., Hall, D., et al. (2023). "Sophia: A scalable stochastic second-order optimizer for language model pre-training". arXiv:2305.14342 — Sophia (3.6 aside).
  • Jordan, K., et al. (2024). "Muon: An optimizer for hidden layers in neural networks". keller-jordan.github.io — Muon (3.6 aside).

Chapter 4 — CNN bridge

Textbooks.

  • Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, §9 (Convolutional Networks). MIT Press. — Convolution, pooling, and CNN architectures, used throughout Cap 4. deeplearningbook.org
  • Prince, S. J. D. (2023). Understanding Deep Learning, §10 (Convolutional Networks). MIT Press. — Receptive fields and equivariance treatment (4.2). udlbook.github.io

Primary papers — CNNs.

  • Hubel, D. H., & Wiesel, T. N. (1962). "Receptive fields, binocular interaction and functional architecture in the cat's visual cortex". Journal of Physiology, 160, 106–154. — Biological origin of local receptive fields (4.2).
  • Fukushima, K. (1980). "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position". Biological Cybernetics, 36, 193–202. — First hierarchical model with weight sharing (4.2).
  • LeCun, Y., Boser, B., Denker, J. S., et al. (1989). "Backpropagation applied to handwritten zip code recognition". Neural Computation, 1(4), 541–551. — First end-to-end trained CNN (4.1).
  • LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). "Gradient-based learning applied to document recognition". Proceedings of the IEEE, 86(11), 2278–2324. — LeNet-5 and the modern conv-pool-conv-pool template (4.1, Appendix 4.B).
  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). "ImageNet classification with deep convolutional neural networks". NeurIPS. — AlexNet; also cited in Cap 2 §2.3.
  • Luo, W., Li, Y., Urtasun, R., & Zemel, R. (2016). "Understanding the effective receptive field in deep convolutional neural networks". NeurIPS. arXiv:1701.04128 — Effective vs theoretical receptive field; Gaussian shape (4.2 §5.3).
  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). "Deep residual learning for image recognition". CVPR. arXiv:1512.03385 — ResNet; deep stacks of small kernels (referenced in 4.2 §5).

Primary papers — convolutional sequence models.

  • van den Oord, A., Dieleman, S., Zen, H., et al. (2016). "WaveNet: A generative model for raw audio". arXiv:1609.03499 — Dilated 1D convolutions for sequences (4.2 Aside, 4.3 §2).
  • Bai, S., Kolter, J. Z., & Koltun, V. (2018). "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling". arXiv:1803.01271 — Temporal convolutional networks; competitive with RNNs but limited at long range (4.3 §2).

Implementation references.

  • Dumoulin, V., & Visin, F. (2016). "A guide to convolution arithmetic for deep learning". arXiv:1603.07285 — Definitive reference for padding, stride, dilation, and transposed convolutions (Appendices 4.A, 4.C).

Chapter 5 — Sequence modeling and embeddings

Textbooks.

  • Jurafsky, D., & Martin, J. H. (2024). Speech and Language Processing (3rd ed., draft). — Canonical reference for tokenisation (5.1), n-grams and smoothing (5.4), and language-modeling fundamentals (5.3). stanford.edu/~jurafsky/slp3
  • Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, §10 (Sequence Modeling: Recurrent and Recursive Nets). MIT Press. — RNN, BPTT, LSTM/GRU (5.5, Appendix 5.A).
  • Prince, S. J. D. (2023). Understanding Deep Learning, §12 (Sequence-to-Sequence Models) and §15 (Recurrent Networks). MIT Press. — Embeddings (5.2), RNN treatment (5.5).
  • Raschka, S. (2024). Build a Large Language Model (From Scratch). Manning. — Chapter 2 on tokenisation (5.1) and the practitioner-level walk-through of BPE.

Primary papers — tokenisation.

  • Gage, P. (1994). "A new algorithm for data compression". The C Users Journal, 12(2), 23–38. — Original BPE algorithm as a compression scheme (5.1).
  • Sennrich, R., Haddow, B., & Birch, A. (2016). "Neural machine translation of rare words with subword units". ACL. arXiv:1508.07909 — Adapted BPE to neural sequence models; the paper that put BPE on the LLM map (5.1).
  • Schuster, M., & Nakajima, K. (2012). "Japanese and Korean voice search". ICASSP. — WordPiece tokeniser (used by BERT) (5.1 §5).
  • Kudo, T., & Richardson, J. (2018). "SentencePiece: A simple and language-independent subword tokenizer and detokenizer for neural text processing". arXiv:1808.06226 — SentencePiece tokeniser (LLaMA, T5) (5.1 §5).
  • Radford, A., Wu, J., Child, R., et al. (2019). "Language models are unsupervised multitask learners". OpenAI technical report. — GPT-2 and the introduction of byte-level BPE (5.1 §5).

Primary papers — embeddings.

  • Firth, J. R. (1957). "A synopsis of linguistic theory, 1930–1955". Studies in Linguistic Analysis, 1–32. — Distributional hypothesis (5.2 §3).
  • Harris, Z. S. (1954). "Distributional structure". Word, 10(2–3), 146–162. — Distributional hypothesis (5.2 §3).
  • Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). "A neural probabilistic language model". JMLR, 3, 1137–1155. — First neural LM with learned word embeddings; the conceptual ancestor of every modern LLM (5.2, 5.4).
  • Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). "Efficient estimation of word representations in vector space". arXiv:1301.3781 — Word2Vec architectures (CBOW, skip-gram) (5.2).
  • Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). "Distributed representations of words and phrases and their compositionality". NeurIPS. arXiv:1310.4546 — Skip-gram with negative sampling; the linear-analogy result (5.2 §4–§5).
  • Pennington, J., Socher, R., & Manning, C. D. (2014). "GloVe: Global vectors for word representation". EMNLP. — Co-occurrence-matrix factorisation as an alternative to word2vec (5.2 §3).
  • Levy, O., & Goldberg, Y. (2014). "Neural word embedding as implicit matrix factorization". NeurIPS. — Proves that SGNS factorises the shifted PMI matrix (5.2 Aside).
  • Press, O., & Wolf, L. (2017). "Using the output embedding to improve language models". EACL. arXiv:1608.05859 — Weight tying between input embedding and output projection (5.2 §6.1).

Primary papers — RNNs and BPTT.

  • Elman, J. L. (1990). "Finding structure in time". Cognitive Science, 14(2), 179–211. — The modern (Elman) RNN (5.5).
  • Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU Munich. — First analysis of vanishing gradients in RNNs (already cited in Cap 2; recalled in 5.5).
  • Bengio, Y., Simard, P., & Frasconi, P. (1994). "Learning long-term dependencies with gradient descent is difficult". IEEE Trans. Neural Networks, 5(2), 157–166. — Formal vanishing/exploding gradient theorem for RNNs (5.5 §4.1).
  • Werbos, P. J. (1990). "Backpropagation through time: What it does and how to do it". Proceedings of the IEEE, 78(10), 1550–1560. — BPTT (5.5 §3).

Primary papers — gated RNNs (Appendix 5.A).

  • Hochreiter, S., & Schmidhuber, J. (1997). "Long short-term memory". Neural Computation, 9(8), 1735–1780. — LSTM cell (Appendix 5.A §3).
  • Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). "Learning to forget: Continual prediction with LSTM". Neural Computation, 12(10), 2451–2471. — Forget gate (Appendix 5.A Aside).
  • Cho, K., van Merriënboer, B., Gulcehre, C., et al. (2014). "Learning phrase representations using RNN encoder–decoder for statistical machine translation". EMNLP. arXiv:1406.1078 — GRU (Appendix 5.A §5).
  • Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). "Empirical evaluation of gated recurrent neural networks on sequence modeling". arXiv:1412.3555 — LSTM/GRU empirical comparison (Appendix 5.A §5).
  • Jozefowicz, R., Zaremba, W., & Sutskever, I. (2015). "An empirical exploration of recurrent network architectures". ICML. — Architecture search over gated RNN variants (Appendix 5.A §5).
  • Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015). "Highway networks". arXiv:1505.00387 — Gated residual feed-forward networks; conceptual bridge from LSTM to ResNet (Appendix 5.A §7).

Lecture series and online resources.

  • Karpathy, A. "Neural Networks: Zero to Hero" lecture series, especially makemore (Lectures 2–5). karpathy.ai/zero-to-hero — Worked examples of bigrams, embeddings, and character-level LMs (5.4).
  • Karpathy, A. (2015). "The unreasonable effectiveness of recurrent neural networks". karpathy.github.io — Cultural-historical reference for LSTM in language modelling (Appendix 5.A Aside).
  • Manning, C., et al. CS224N: Natural Language Processing with Deep Learning. Stanford. — Lectures on subword tokenisation and language modelling (5.1, 5.3).

Chapter 6 — Attention and the Transformer

Textbooks.

  • Prince, S. J. D. (2023). Understanding Deep Learning, §12 (Sequence-to-Sequence Models) and §13 (Transformers). MIT Press. — Attention as soft lookup (6.1), scaled dot-product attention (6.2), multi-head attention (6.4).
  • Raschka, S. (2024). Build a Large Language Model (From Scratch), Chapters 3–4. Manning. — Self-attention implementation (6.3, 6.4), causal masking (6.5), pre-norm block (6.10–6.11).
  • Bishop, C. M., & Bishop, H. (2024). Deep Learning: Foundations and Concepts, §12 (Transformers). Springer. — Concise modern treatment of all attention components.

Primary papers — core attention and Transformer.

  • Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). "Attention is all you need". NeurIPS. arXiv:1706.03762 — The Transformer paper. Scaled dot-product attention (6.2), multi-head attention (6.4), encoder–decoder cross-attention (6.3), sinusoidal positional encoding (Appendix 6.A), the FFN sub-layer (6.8), post-LN block diagram (6.10), full architecture read (6.11).
  • Bahdanau, D., Cho, K., & Bengio, Y. (2014). "Neural machine translation by jointly learning to align and translate". arXiv:1409.0473 — First "neural attention" paper; additive attention as a precursor to scaled dot-product (6.1, 6.2).
  • Luong, M.-T., Pham, H., & Manning, C. D. (2015). "Effective approaches to attention-based neural machine translation". EMNLP. arXiv:1508.04025 — Dot-product attention variants (6.2).
  • Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). "Improving language understanding by generative pre-training" (GPT-1). OpenAI technical report. — Decoder-only Transformer with causal self-attention (6.3, 6.5); first use of learned PE (Appendix 6.B).
  • Radford, A., Wu, J., Child, R., et al. (2019). "Language models are unsupervised multitask learners" (GPT-2). OpenAI technical report. — Learned absolute positional encoding (Appendix 6.B), pre-LN variant (6.10), the architectural skeleton of every modern decoder-only LLM (6.11).

Primary papers — positional encodings.

  • Shaw, P., Uszkoreit, J., & Vaswani, A. (2018). "Self-attention with relative position representations". NAACL. arXiv:1803.02155 — First relative positional encoding (6.6 §4.2).
  • Raffel, C., Shazeer, N., Roberts, A., et al. (2020). "Exploring the limits of transfer learning with a unified text-to-text transformer" (T5). JMLR. arXiv:1910.10683 — Bucketed relative positional bias (6.6 §4.2).
  • Su, J., Lu, Y., Pan, S., et al. (2021). "RoFormer: Enhanced transformer with rotary position embedding". arXiv:2104.09864 — Rotary positional embedding (RoPE) (6.7).
  • Press, O., Smith, N. A., & Lewis, M. (2021). "Train short, test long: Attention with linear biases enables input length extrapolation" (ALiBi). arXiv:2108.12409 — Linear-distance bias inside softmax; length extrapolation (6.13 §3).
  • Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). "Extending context window of large language models via positional interpolation". arXiv:2306.15595 — Linear interpolation of RoPE frequencies for context extension (6.13 §4.1).
  • Peng, B., Quesnelle, J., Fan, H., & Shippole, E. (2023). "YaRN: Efficient context window extension of large language models". arXiv:2309.00071 — Piecewise RoPE scaling (NTK-aware + linear-interpolation hybrid) (6.13 §4.3).

Primary papers — residual connections and normalisation placement.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2015). "Deep residual learning for image recognition" (ResNet). arXiv:1512.03385 — Identity skip connections; the architectural innovation imported wholesale into the Transformer (6.9).
  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). "Identity mappings in deep residual networks". arXiv:1603.05027 — Pre-activation ResNet; the structural blueprint behind pre-LN Transformer (6.10 §6).
  • Veit, A., Wilber, M. J., & Belongie, S. (2016). "Residual networks behave like ensembles of relatively shallow networks". arXiv:1605.06431 — Implicit-ensemble view of residual networks (6.9 Aside).
  • Zhang, H., Dauphin, Y. N., & Ma, T. (2019). "Fixup initialization: Residual learning without normalization". arXiv:1901.09321 — Identity-initialisation tricks for very deep residual stacks (6.9 §4).
  • Wang, Q., Li, B., Xiao, T., et al. (2019). "Improving deep transformer with depth-scaled initialization and merged attention" (a.k.a. learning deep Transformers). arXiv:1908.11365 — Early analysis of post-LN instability; motivation for pre-LN (6.10 §4.2).
  • Xiong, R., Yang, Y., He, D., et al. (2020). "On layer normalization in the Transformer architecture". arXiv:2002.04745 — Theoretical analysis: post-LN gradient grows as √L, pre-LN is O(1) (6.10 §5).
  • Wang, H., Ma, S., Dong, L., et al. (2022). "DeepNet: Scaling Transformers to 1000 layers" (DeepNorm). arXiv:2203.00555 — Stable post-norm variant via residual scaling (6.10 §8).
  • Shleifer, S., Weston, J., & Ott, M. (2021). "NormFormer: Improved transformer pretraining with extra normalization". arXiv:2110.09456 — Additional norm placements for deeper models (6.10 §8).

Primary papers — long-context attention.

  • Beltagy, I., Peters, M. E., & Cohan, A. (2020). "Longformer: The long-document transformer". arXiv:2004.05150 — Sliding-window + global attention with dilations (6.13 §2).
  • Jiang, A. Q., Sablayrolles, A., Mensch, A., et al. (2023). "Mistral 7B". arXiv:2310.06825 — Sliding-window attention at scale; pairs with attention sinks (6.13 §2).
  • Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2023). "Efficient streaming language models with attention sinks" (StreamingLLM). arXiv:2309.17453 — Attention sinks for constant-memory streaming (6.13 §2.2).

Primary papers — KV-cache and inference-side optimisations.

  • Shazeer, N. (2019). "Fast transformer decoding: One write-head is all you need" (Multi-Query Attention). arXiv:1911.02150 — MQA: share K, V across heads to shrink the KV cache (6.14 §5.1).
  • Ainslie, J., Lee-Thorp, J., de Jong, M., et al. (2023). "GQA: Training generalized multi-query transformer models from multi-head checkpoints". arXiv:2305.13245 — Grouped-query attention; the modern default in LLaMA-2-70B, LLaMA-3, Mistral, Qwen (6.14 §5.1; full treatment Cap 8.3).
  • DeepSeek-AI (2024). "DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model" (MLA). arXiv:2405.04434 — Multi-head latent attention; low-rank K, V compression (6.14 §5.2; full treatment Cap 8.5).
  • Kwon, W., Li, Z., Zhuang, S., et al. (2023). "Efficient memory management for large language model serving with PagedAttention" (vLLM). arXiv:2309.06180 — OS-style paging for the KV cache (6.14 §6; full treatment Cap 10.8).
  • Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., & Chun, B.-G. (2022). "Orca: A distributed serving system for transformer-based generative models". OSDI. — Continuous batching (6.14 §6).

Primary papers — interpretability of heads and FFN.

  • Olsson, C., Elhage, N., Nanda, N., et al. (2022). "In-context learning and induction heads". Anthropic. transformer-circuits.pub — Induction heads as the mechanism behind in-context learning (6.4 §4); residual-stream view (6.11 §4).
  • Geva, M., Schuster, R., Berant, J., & Levy, O. (2021). "Transformer feed-forward layers are key-value memories". EMNLP. arXiv:2012.14913 — FFN as parametric associative memory (6.8 §6).
  • Shazeer, N. (2020). "GLU variants improve Transformer". arXiv:2002.05202 — SwiGLU / GeGLU; modern FFN variant (6.8 Aside; full treatment in Cap 8.4).

Lecture series, online resources, and reference implementations.

  • 3Blue1Brown. "But what is a GPT? Visual intro to Transformers" and "Attention in transformers, step-by-step". 3blue1brown.com — Visual introduction to attention (6.1, 6.2).
  • Karpathy, A. "Let's build GPT: from scratch, in code, spelled out". YouTube; part of the Zero-to-Hero series at karpathy.ai — Live coding of a Transformer; pairs directly with 6.15–6.16.
  • Karpathy, A. nanoGPT. github.com/karpathy/nanoGPT — Reference implementation walked through line-by-line in 6.16.
  • Alammar, J. (2018). "The illustrated Transformer". jalammar.github.io — Canonical visual walkthrough (6.4, 6.8).

Chapter 7 — Training language models

Textbooks and lecture series.

  • Stanford CS336 ("Language Modeling from Scratch", 2024 onwards). stanford-cs336.github.io — End-to-end LLM pretraining course; data, parallelism, scaling.
  • Raschka, S. (2024). Build a Large Language Model (From Scratch), Chapters 5–6. Manning. — End-to-end training of a small LLM.
  • Karpathy, A. "Let's build the GPT tokenizer". YouTube (2024); paired with minbpe. — Implements byte-level BPE from scratch (7.2).

Primary papers — pretraining data.

  • Raffel, C., Shazeer, N., Roberts, A., et al. (2020). "Exploring the limits of transfer learning with a unified text-to-text transformer" (T5). JMLR. arXiv:1910.10683 — C4 corpus (7.1 §4.1).
  • Gao, L., Biderman, S., Black, S., et al. (2020). "The Pile: An 800GB dataset of diverse text for language modeling". arXiv:2101.00027 — 22-source mixed corpus (7.1 §4.2).
  • Together AI (2023). "RedPajama: An open source recipe to reproduce LLaMA training dataset". together.ai — Open reproduction of LLaMA-1 data (7.1 §4.3).
  • Li, J., Fang, A., Smyrnis, G., et al. (2024). "DataComp-LM: In search of the next generation of training sets for language models" (DCLM). arXiv:2406.11794 — Systematic data-quality experiments; DCLM-Baseline (7.1 §4.4, 7.5).
  • Penedo, G., Kydlíček, H., Cappelli, A., et al. (2024). "The FineWeb Datasets: Decanting the web for the finest text data at scale". arXiv:2406.17557; HuggingFace blog. — FineWeb and FineWeb-Edu (7.1 §4.5, 7.5).
  • Rae, J. W., Borgeaud, S., Cai, T., et al. (2021). "Scaling language models: Methods, analysis & insights from training Gopher" (MassiveText). arXiv:2112.11446 — Heuristic data-filtering rules (7.1 §3.3).
  • Wenzek, G., Lachaux, M.-A., Conneau, A., et al. (2020). "CCNet: Extracting high quality monolingual datasets from web crawl data". arXiv:1911.00359 — Classifier-based filtering pipeline (7.1 §3.4).
  • Lee, K., Ippolito, D., Nystrom, A., et al. (2022). "Deduplicating training data makes language models better". arXiv:2107.06499 — MinHash deduplication (7.1 §6).
  • Villalobos, P., Sevilla, J., Heim, L., et al. (2022). "Will we run out of data? An analysis of the limits of scaling datasets in machine learning". arXiv:2211.04325 — The "data wall" argument (7.1 §5).
  • Muennighoff, N., Rush, A. M., Barak, B., et al. (2023). "Scaling data-constrained language models". arXiv:2305.16264 — Repeating high-quality data up to 4× (7.1 §5, 7.5).
  • Welbl, J., Glaese, A., Uesato, J., et al. (2021). "Challenges in detoxifying language models". arXiv:2109.07445 — Toxicity filtering trade-offs (7.1 §3.7).
  • Barbaresi, A. (2021). "Trafilatura: a web scraping library and command-line tool for text discovery and extraction". ACL System Demonstrations. aclanthology.org/2021.acl-demo.15 — Reference HTML→text extractor (7.1 §3.1).

Primary papers — tokenization.

  • Sennrich, R., Haddow, B., & Birch, A. (2016). "Neural machine translation of rare words with subword units" (BPE). arXiv:1508.07909 — BPE adapted from Gage 1994 for NMT (7.2).
  • Kudo, T., & Richardson, J. (2018). "SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing". arXiv:1808.06226 — SentencePiece (BPE-mode and unigram-mode) (7.2 §5.4).
  • Gage, P. (1994). "A new algorithm for data compression". C Users Journal, 12(2), 23–38. — The original byte-pair encoding compression algorithm (7.2 §3.4).

Primary papers — precision and numerical training.

  • Micikevicius, P., Narang, S., Alben, J., et al. (2017). "Mixed precision training". arXiv:1710.03740 — fp16 mixed precision with loss scaling (7.4 §3, §4).
  • Wang, S., & Kanwar, P. (2019). "BFloat16: The secret to high performance on Cloud TPUs". Google Cloud blog. cloud.google.com — bf16 design rationale (7.4 §2.1).
  • Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). "LLM.int8(): 8-bit matrix multiplication for transformers at scale". arXiv:2208.07339 — Outlier-aware quantisation; structural activation outliers (7.4 §6.2; full treatment Cap 10.6).

Primary papers — data sequencing.

  • Xie, S. M., Pham, H., Dong, X., et al. (2023). "DoReMi: Optimizing data mixtures speeds up language model pretraining". arXiv:2305.10429 — Worst-domain-perplexity optimisation for data mixtures (7.5 §2.2).
  • Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). "Curriculum learning". ICML. dl.acm.org/doi/10.1145/1553374.1553380 — Original curriculum-learning paper (7.5 §3).
  • Hu, S., Tu, Y., Han, X., et al. (2024). "MiniCPM: Unveiling the potential of small language models with scalable training strategies". arXiv:2404.06395 — WSD schedule + late-stage annealing recipe (7.5 §4).
  • Abdin, M., Aneja, J., Awadalla, H., et al. (2024). "Phi-3 technical report". arXiv:2404.14219 — Synthetic/textbook-heavy data approach (7.5 §6 Aside).
  • Dubey, A., Jauhri, A., Pandey, A., et al. (2024). "The Llama 3 herd of models". arXiv:2407.21783 — LLaMA-3 data-mixing-in-late-stages documentation (7.1 §4.6, 7.5).

Primary papers — memory and compute engineering.

  • Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016). "Training deep nets with sublinear memory cost". arXiv:1604.06174 — Activation checkpointing / gradient checkpointing (7.7 §4).
  • Korthikanti, V., Casper, J., Lym, S., et al. (2022). "Reducing activation recomputation in large transformer models". arXiv:2205.05198 — Selective recomputation in Megatron-LM (7.7 §7).
  • Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). "FlashAttention: Fast and memory-efficient exact attention with IO-awareness". arXiv:2205.14135 — Tiled attention kernel; in-SRAM softmax (7.7 §7.1; full treatment Cap 10.5).

Primary papers — distributed training.

  • Patarasuk, P., & Yuan, X. (2009). "Bandwidth optimal all-reduce algorithms for clusters of workstations". J. Parallel and Distributed Computing. sciencedirect.com — Ring all-reduce algorithm (7.8 §3.2).
  • McCandlish, S., Kaplan, J., Amodei, D., et al. (2018). "An empirical model of large-batch training". arXiv:1812.06162 — Critical batch size (7.8 §6, 7.11 §5).
  • Shoeybi, M., Patwary, M., Puri, R., et al. (2019). "Megatron-LM: Training multi-billion parameter language models using model parallelism". arXiv:1909.08053 — Tensor parallelism (column/row sharding of FFN and MHA) (7.9 §2).
  • Korthikanti, V., Casper, J., Lym, S., et al. (2022). "Reducing activation recomputation in large transformer models". arXiv:2205.05198 — Sequence parallelism + selective activation recomputation (7.9 §2.5; also 7.7 §7).
  • Huang, Y., Cheng, Y., Bapna, A., et al. (2018). "GPipe: Efficient training of giant neural networks using pipeline parallelism". arXiv:1811.06965 — Microbatch-based pipeline parallelism; bubble fraction (7.9 §3).
  • Narayanan, D., Harlap, A., Phanishayee, A., et al. (2019). "PipeDream: Generalized pipeline parallelism for DNN training". arXiv:1806.03377 — 1F1B pipeline schedule (7.9 §3.3).
  • Narayanan, D., Shoeybi, M., Casper, J., et al. (2021). "Efficient large-scale language model training on GPU clusters using Megatron-LM". arXiv:2104.04473 — Interleaved pipeline + TP × PP × DP composition (7.9 §3.4).
  • Liu, H., Zaharia, M., & Abbeel, P. (2023). "Ring Attention with blockwise transformers for near-infinite context". arXiv:2310.01889 — Context parallelism for 1M+ context training (7.9 §6, 7.13 §5.1).
  • Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2020). "ZeRO: Memory optimizations toward training trillion parameter models". arXiv:1910.02054 — Three-stage memory sharding (ZeRO-1/2/3 / FSDP) (7.10).
  • Ren, J., Rajbhandari, S., Aminabadi, R. Y., et al. (2021). "ZeRO-Offload: Democratizing billion-scale model training". arXiv:2101.06840 — Offloading Adam state to CPU memory (7.10 §8).
  • Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., & He, Y. (2021). "ZeRO-Infinity: Breaking the GPU memory wall for extreme scale deep learning". arXiv:2104.07857 — NVMe offload extending ZeRO (7.10 §8).

Primary papers — scaling laws.

  • Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). "Scaling laws for neural language models". arXiv:2001.08361 — Power-law scaling of loss in N,D,CN, D, C; the framework still in use (7.11).
  • Henighan, T., Kaplan, J., Katz, M., et al. (2020). "Scaling laws for autoregressive generative modeling". arXiv:2010.14701 — Extension to images, video, code modalities (7.11).
  • Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). "Training compute-optimal large language models" (Chinchilla). arXiv:2203.15556 — The D/N≈20D/N \approx 20 correction (7.12).
  • Sardana, N., & Frankle, J. (2024). "Beyond Chinchilla-optimal: Accounting for inference in language model scaling laws". arXiv:2401.00448 — Inference-aware over-training derivation (7.12 §5).

Primary papers — long-context training (training-side complement to Cap 6.13 references).

  • Liu, N. F., Lin, K., Hewitt, J., et al. (2023). "Lost in the middle: How language models use long contexts". arXiv:2307.03172 — The mid-context performance dip failure mode (7.13 §4.3).
  • Note: The other long-context RoPE-scaling papers (Chen et al. 2023 position interpolation; YaRN by Peng et al. 2023; Mistral sliding-window; ALiBi by Press et al. 2021; attention sinks by Xiao et al. 2023; FlashAttention by Dao et al. 2022) are listed under Chapter 6's positional-encodings and long-context sections, and are referenced by 7.13 in their training-side application.

Primary papers — GPT-3 and the few-shot story.

  • Brown, T. B., Mann, B., Ryder, N., et al. (2020). "Language models are few-shot learners" (GPT-3). arXiv:2005.14165 — The 175B-parameter pretrained LM; few-shot prompting at scale (7.14).
  • Child, R., Gray, S., Radford, A., & Sutskever, I. (2019). "Generating long sequences with sparse transformers". arXiv:1904.10509 — Sparse Transformer's alternating sparse-dense pattern; used in GPT-3 (7.14 §3.1).
  • Radford, A., Wu, J., Child, R., et al. (2019). "Language models are unsupervised multitask learners" (GPT-2). OpenAI technical report. — Pre-LN block, modified init, reversible tokenization referenced by GPT-3 §2.1 (7.14 §3.1).

Chapter 8 — Modern LLM architectures

Primary papers — encoder-only.

  • Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). "BERT: Pre-training of deep bidirectional transformers for language understanding". NAACL. arXiv:1810.04805 — BERT, masked language modeling (8.2).
  • Liu, Y., Ott, M., Goyal, N., et al. (2019). "RoBERTa: A robustly optimized BERT pretraining approach". arXiv:1907.11692 — Drops NSP, longer training (8.2).
  • Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020). "ELECTRA: Pre-training text encoders as discriminators rather than generators". arXiv:2003.10555 — Replaced-token-detection objective; denser signal than MLM (8.2 §5).
  • Wang, L., Yang, N., Huang, X., et al. (2022). "Text embeddings by weakly-supervised contrastive pre-training" (E5). arXiv:2212.03533 — Bidirectional-encoder embedder pretraining (8.2 §4).
  • Wang, T., Roberts, A., Hesslow, D., et al. (2022). "What language model architecture and pretraining objective work best for zero-shot generalization?". arXiv:2204.05832 — Decoder-only-causal vs encoder-only-MLM vs encoder-decoder-span comparison (8.1 §4, 8.2 §3).

Primary papers — KV-cache attention variants.

  • Shazeer, N. (2019). "Fast transformer decoding: One write-head is all you need" (MQA). arXiv:1911.02150 — Multi-query attention (8.3).
  • Ainslie, J., Lee-Thorp, J., de Jong, M., et al. (2023). "GQA: Training generalized multi-query transformer models from multi-head checkpoints". arXiv:2305.13245 — Grouped-query attention and the mean-pool uptraining recipe (8.3).
  • DeepSeek-AI (2024). "DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model". arXiv:2405.04434 — Multi-head Latent Attention (MLA) with decoupled RoPE (8.5); also primary source for fine-grained MoE (8.6).
  • DeepSeek-AI (2024). "DeepSeek-V3 technical report". arXiv:2412.19437 — V3 architecture: MLA + 256-expert fine-grained MoE + auxiliary-loss-free balancing (8.5, 8.6 §6).
  • DeepSeek-AI (2025). "DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning". arXiv:2501.12948 — R1 architecture inherits V3's MLA + MoE (referenced in 8.5; primary citation in Cap 9.7).

Primary papers — FFN variants.

  • Dauphin, Y. N., Fan, A., Auli, M., & Grangier, D. (2017). "Language modeling with gated convolutional networks". arXiv:1612.08083 — Gated linear units (GLU); ancestor of SwiGLU (8.4 §2).
  • Shazeer, N. (2020). "GLU variants improve Transformer". arXiv:2002.05202 — SwiGLU, GeGLU empirical sweep (8.4); also already cited in 2.3 and 6.8.
  • Chowdhery, A., Narang, S., Devlin, J., et al. (2022). "PaLM: Scaling language modeling with pathways". arXiv:2204.02311 — Adopted GeGLU and dropped FFN biases; reference for the "no biases" convention (8.4 §5 Aside).
  • Touvron, H., Lavril, T., Izacard, G., et al. (2023). "LLaMA: Open and efficient foundation language models". arXiv:2302.13971 — Architectural bundle: RMSNorm + RoPE + SwiGLU; defaults for dff=(8/3)dd_\text{ff} = (8/3)d (8.4 §3, 8.4 §5).

Primary papers — Mixture of Experts.

  • Shazeer, N., Mirhoseini, A., Maziarz, K., et al. (2017). "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer". arXiv:1701.06538 — Original deep MoE with top-kk gating (8.6 §2).
  • Lepikhin, D., Lee, H., Xu, Y., et al. (2020). "GShard: Scaling giant models with conditional computation and automatic sharding". arXiv:2006.16668 — Capacity factor, auxiliary load-balance loss, expert parallelism (8.6 §3, §6).
  • Fedus, W., Zoph, B., & Shazeer, N. (2021). "Switch Transformer: Scaling to trillion parameter models with simple and efficient sparsity". arXiv:2101.03961 — Top-1 routing; standard auxiliary loss formulation (8.6 §3.1).
  • Dai, D., Deng, C., Zhao, C., et al. (2024). "DeepSeek-MoE: Towards ultimate expert specialization in mixture-of-experts language models". arXiv:2401.06066 — Fine-grained experts + shared experts (8.6 §4).
  • Jiang, A. Q., Sablayrolles, A., Roux, A., et al. (2024). "Mixtral of experts". arXiv:2401.04088 — Mixtral-8x7B; production MoE LLM (8.6 §5).
  • Wang, L., Gao, H., Zhao, C., Sun, X., & Dai, D. (2024). "Auxiliary-loss-free load balancing strategy for mixture-of-experts" (DeepSeek-V3 routing). arXiv:2408.15664 — Control-loop-based balance without auxiliary loss (8.6 §6 Aside).

Primary papers — State Space Models.

  • Kalman, R. E. (1960). "A new approach to linear filtering and prediction problems". Trans. ASME—J. Basic Eng., 82, 35–45. — Canonical state-space formulation of linear filtering, ancestor of all SSM architectures (8.7 §2).
  • Gu, A., Goel, K., & Ré, C. (2022). "Efficiently modeling long sequences with structured state spaces" (S4). arXiv:2111.00396 — Structured (diagonal-plus-low-rank) AA for efficient long-sequence SSM (8.7 §3.1).
  • Gu, A., & Dao, T. (2023). "Mamba: Linear-time sequence modeling with selective state spaces". arXiv:2312.00752 — Selective SSM with input-dependent Δ,B,C\Delta, B, C; hardware-aware parallel scan (8.7 §4).
  • Dao, T., & Gu, A. (2024). "Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality" (Mamba-2). arXiv:2405.21060 — SSM-attention duality; Mamba-2 architecture (8.7 §5.1).

Primary papers — Hybrid architectures.

  • Lieber, O., Lenz, B., Bata, H., et al. (2024). "Jamba: A hybrid Transformer-Mamba language model". arXiv:2403.19887 — 1 attention layer per 8; MoE in alternating layers (8.8 §2).
  • Ren, L., Liu, Y., Lu, Y., et al. (2024). "Samba: Simple hybrid state space models for efficient unlimited context language modeling". arXiv:2406.07522 — Mamba + sliding-window attention; constant-cache long-context (8.8 §3).
  • Pióro, M., Ciebiera, K., Krol, K., et al. (2024). "MoE-Mamba: Efficient selective state space models with mixture of experts". arXiv:2401.04081 — MoE FFN in Mamba blocks (8.8 §4).
  • Dong, X., Fu, Y., Diao, S., et al. (2024). "Hymba: A hybrid-head architecture for small language models" (NVIDIA). arXiv:2411.13676 — Parallel attention + SSM in the same block (8.8 §4).
  • Glorioso, P., Anthony, Q., Tokpanov, Y., et al. (2024). "Zamba: A compact 7B SSM hybrid model" (Zyphra). arXiv:2405.16712 — Shared attention block across Mamba layers (8.8 §4).
  • De, S., Smith, S. L., Fernando, A., et al. (2024). "Griffin: Mixing gated linear recurrences with local attention for efficient language models" (DeepMind). arXiv:2402.19427 — Recurrent linear attention + local attention hybrid (8.8 §4).

Primary papers — Encoder-decoder (Appendix 8.A).

  • Raffel, C., Shazeer, N., Roberts, A., et al. (2020). "Exploring the limits of transfer learning with a unified text-to-text transformer" (T5). JMLR. arXiv:1910.10683 — Encoder-decoder with span corruption (Appendix 8.A); also cited in Ch 6 (T5 relative bias) and Ch 7 (C4 corpus).
  • Tay, Y., Dehghani, M., Tran, V. Q., et al. (2022). "UL2: Unifying language learning paradigms". arXiv:2205.05131 — Mixture of denoisers; T5 successor (Appendix 8.A §5 Aside).

Chapter 9 — Alignment: SFT, RLHF, DPO, GRPO

Primary papers — instruction tuning and RLHF.

  • Ouyang, L., Wu, J., Jiang, X., et al. (2022). "Training language models to follow instructions with human feedback" (InstructGPT). arXiv:2203.02155 — Canonical SFT + RM + PPO pipeline (9.1, 9.2, 9.8); per-token KL implementation (9.9); PPO-ptx mixing (9.8 §5.3).
  • Christiano, P., Leike, J., Brown, T., et al. (2017). "Deep reinforcement learning from human preferences". arXiv:1706.03741 — Original RLHF in deep RL; the algorithm InstructGPT inherits (9.3).
  • Stiennon, N., Ouyang, L., Wu, J., et al. (2020). "Learning to summarize from human feedback". arXiv:2009.01325 — RLHF applied to summarisation; bridges Christiano 2017 to InstructGPT (9.3).

Primary papers — preference modelling.

  • Bradley, R. A., & Terry, M. E. (1952). "Rank analysis of incomplete block designs: I. The method of paired comparisons". Biometrika, 39, 324–345. — Bradley-Terry pairwise-preference model (9.4).
  • Luce, R. D. (1959). Individual Choice Behavior: A Theoretical Analysis. Wiley. — Plackett-Luce; generalisation to multi-way comparisons (9.4 §6).

Primary papers — policy gradient and PPO.

  • Williams, R. J. (1992). "Simple statistical gradient-following algorithms for connectionist reinforcement learning". Machine Learning, 8, 229–256. — Original REINFORCE (9.5).
  • Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. Chapter 13. — Policy-gradient methods, baselines (9.5).
  • Schulman, J., Levine, S., Abbeel, P., Jordan, M., & Moritz, P. (2015). "Trust region policy optimization" (TRPO). arXiv:1502.05477 — Constrained policy update; the precursor PPO simplifies (9.6, 9.9 §4.3).
  • Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). "Proximal policy optimization algorithms" (PPO). arXiv:1707.06347 — Clipped surrogate objective; the workhorse of RLHF (9.6).
  • Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). "High-dimensional continuous control using generalized advantage estimation" (GAE). arXiv:1506.02438 — GAE; used by PPO inside RLHF (9.6 §5).

Primary papers — GRPO and group-based methods.

  • Shao, Z., Wang, P., Zhu, Q., et al. (2024). "DeepSeekMath: Pushing the limits of mathematical reasoning in open language models". arXiv:2402.03300 — Group Relative Policy Optimization (GRPO) (9.7).
  • DeepSeek-AI (2025). "DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning". arXiv:2501.12948 — GRPO at frontier reasoning scale; R1-Zero pure-RL training (9.7, 9.12).
  • Kool, W., van Hoof, H., & Welling, M. (2019). "Buy 4 REINFORCE samples, get a baseline for free!". ICLR Workshop. — Leave-one-out baseline (RLOO precursor) (9.11 §7.2).

Primary papers — Direct Preference Optimization.

  • Rafailov, R., Sharma, A., Mitchell, E., et al. (2023). "Direct preference optimization: Your language model is secretly a reward model" (DPO). arXiv:2305.18290 — DPO derivation (9.10).
  • Pang, R. Y., Yuan, W., Cho, K., et al. (2024). "Iterative reasoning preference optimization". arXiv:2404.19733 — Iterative DPO recipe (9.11).
  • Chen, Z., Deng, Y., Yuan, H., et al. (2024). "Self-play fine-tuning converts weak language models to strong language models" (SPIN). arXiv:2401.01335 — Iterative DPO with SFT data as ycy_c (9.11 §7.3).
  • Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). "Tülu 3: Pushing frontiers in open language model post-training". arXiv:2411.15124 — Iterative DPO with mixed labellers; open recipe (9.11 §6).

Primary papers — Process Reward Models.

  • Lightman, H., Kosaraju, V., Burda, Y., et al. (2023). "Let's verify step by step". arXiv:2305.20050 — PRM800K dataset; foundational PRM paper (9.12).
  • Wang, P., Li, L., Shao, Z., et al. (2024). "Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations". arXiv:2312.08935 — Outcome-conditioned synthetic PRM labels (9.12 §6.2).

Primary papers — RLAIF and Constitutional AI.

  • Bai, Y., Kadavath, S., Kundu, S., et al. (2022). "Constitutional AI: Harmlessness from AI feedback" (CAI). arXiv:2212.08073 — Anthropic's CAI recipe; critique-and-revise + AI-judged preferences (9.13).
  • Lee, H., Phatale, S., Mansoor, H., et al. (2023). "RLAIF: Scaling reinforcement learning from human feedback with AI feedback". arXiv:2309.00267 — Empirical comparison RLHF vs RLAIF (9.13 §2.2).

Primary papers — Reward hacking and over-optimization.

  • Gao, L., Schulman, J., & Hilton, J. (2022). "Scaling laws for reward model overoptimization". arXiv:2210.10760 — Scaling laws of over-optimisation; the α1KL−α2KL\alpha_1 \sqrt{\mathrm{KL}} - \alpha_2 \mathrm{KL} curve (9.14).
  • Coste, T., Anwar, U., Kirk, R., & Krueger, D. (2024). "Reward model ensembles help mitigate overoptimization". arXiv:2310.02743 — RM ensembles to reduce hacking (9.14 §4.4).

Primary papers — Inference-time alignment.

  • Cobbe, K., Kosaraju, V., Bavarian, M., et al. (2021). "Training verifiers to solve math word problems" (GSM8K + verifiers). arXiv:2110.14168 — Verifier-based Best-of-N (9.15 §2.3).
  • Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). "Scaling LLM test-time compute optimally can be more effective than scaling model parameters". arXiv:2408.03314 — Inference-time scaling laws (9.15 §6.2).

Chapter 10 — Inference and decoding

Textbooks / canonical references.

  • Jurafsky, D., & Martin, J. H. (2025, draft 3rd ed.). Speech and Language Processing. Ch 9 covers greedy/beam decoding (10.1) and modern sampling (10.2). stanford.edu/~jurafsky/slp3

Primary papers — sampling and decoding.

  • Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019). "The curious case of neural text degeneration". arXiv:1904.09751 — Beam search curse + nucleus (top-pp) sampling (10.1, 10.2).
  • Wu, Y., Schuster, M., Chen, Z., et al. (2016). "Google's neural machine translation system: Bridging the gap between human and machine translation". arXiv:1609.08144 — Length-normalisation in beam search (10.1).
  • Keskar, N. S., McCann, B., Varshney, L. R., et al. (2019). "CTRL: A conditional transformer language model for controllable generation". arXiv:1909.05858 — Repetition penalty (10.2).
  • Nguyen, M., Baker, A., Neo, C., et al. (2024). "Turning up the heat: Min-pp sampling for creative and coherent LLM outputs". arXiv:2407.01082 — Min-pp sampling (10.2).

Primary papers — speculative decoding.

  • Leviathan, Y., Kalman, M., & Matias, Y. (2023). "Fast inference from transformers via speculative decoding". arXiv:2211.17192 — Speculative decoding protocol and rejection-sampling correctness (10.4).
  • Chen, C., Borgeaud, S., Irving, G., et al. (2023). "Accelerating large language model decoding with speculative sampling". arXiv:2302.01318 — Parallel discovery of the same technique (10.4).
  • Cai, T., Li, Y., Geng, Z., et al. (2024). "Medusa: Simple LLM inference acceleration framework with multiple decoding heads". arXiv:2401.10774 — Medusa head-based drafting (10.4).
  • Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024). "EAGLE: Speculative sampling requires rethinking feature uncertainty". arXiv:2401.15077 — EAGLE draft architecture (10.4).

Primary papers — FlashAttention.

  • Dao, T., Fu, D., Ermon, S., Rudra, A., & Ré, C. (2022). "FlashAttention: Fast and memory-efficient exact attention with IO-awareness". arXiv:2205.14135 — FlashAttention-1 (10.5).
  • Dao, T. (2023). "FlashAttention-2: Faster attention with better parallelism and work partitioning". arXiv:2307.08691 — FlashAttention-2 (10.5).
  • Shah, J., Bikshandi, G., Zhang, Y., et al. (2024). "FlashAttention-3: Fast and accurate attention with asynchrony and low-precision". arXiv:2407.08608 — FlashAttention-3 on H100 (10.5).

Primary papers — quantisation.

  • Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). "GPTQ: Accurate post-training quantization for generative pre-trained transformers". arXiv:2210.17323 — GPTQ (10.6).
  • Lin, J., Tang, J., Tang, H., et al. (2023). "AWQ: Activation-aware weight quantization for LLM compression and acceleration". arXiv:2306.00978 — AWQ (10.6).
  • Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). "LLM.int8(): 8-bit matrix multiplication for transformers at scale". arXiv:2208.07339 — The outlier problem and INT8 with mixed-precision fallback (10.6).
  • Xiao, G., Lin, J., Seznec, M., et al. (2022). "SmoothQuant: Accurate and efficient post-training quantization for large language models". arXiv:2211.10438 — Activation-weight scaling for INT8-everywhere inference (10.6).
  • Liu, Z., Yuan, J., Jin, H., et al. (2024). "KIVI: A tuning-free asymmetric 2bit quantization for KV cache". arXiv:2402.02750 — KV-cache quantisation recipe (10.6).

Primary papers — test-time scaling and reasoning models.

  • Wei, J., Wang, X., Schuurmans, D., et al. (2022). "Chain-of-thought prompting elicits reasoning in large language models". arXiv:2201.11903 — Chain-of-thought (10.7, 10.9).
  • Wang, X., Wei, J., Schuurmans, D., et al. (2022). "Self-consistency improves chain of thought reasoning in language models". arXiv:2203.11171 — Self-consistency / majority vote over reasoning chains (10.7, 10.9).
  • Zelikman, E., Wu, Y., Mu, J., & Goodman, N. (2022). "STaR: Bootstrapping reasoning with reasoning". arXiv:2203.14465 — Self-bootstrapping reasoning training (10.7).
  • DeepSeek-AI (2025). "DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning". arXiv:2501.12948 — R1 / R1-Zero reasoning-model recipe (10.7).
  • Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). "Scaling LLM test-time compute optimally can be more effective than scaling model parameters". arXiv:2408.03314 — Inference-time scaling laws (10.7).
  • Lanham, T., Chen, A., Radhakrishnan, A., et al. (2023). "Measuring faithfulness in chain-of-thought reasoning". arXiv:2307.13702 — Faithfulness of CoT (10.7, 11.9).
  • Turpin, M., Michael, J., Perez, E., & Bowman, S. (2023). "Language models don't always say what they think". arXiv:2305.04388 — Performative vs faithful chains (10.7, 11.9).

Primary papers — serving infrastructure.

  • Yu, G.-I., Jeong, J. S., Kim, G.-W., et al. (2022). "Orca: A distributed serving system for transformer-based generative models". OSDI 2022. usenix.org — Iteration-level scheduling / continuous batching (10.8).
  • Kwon, W., Li, Z., Zhuang, S., et al. (2023). "Efficient memory management for large language model serving with PagedAttention". arXiv:2309.06180 — vLLM and PagedAttention (10.8).

Primary papers — agentic decoding.

  • Yao, S., Yu, D., Zhao, J., et al. (2023). "Tree of Thoughts: Deliberate problem solving with large language models". arXiv:2305.10601 — ToT (10.9).
  • Yao, S., Zhao, J., Yu, D., et al. (2022). "ReAct: Synergizing reasoning and acting in language models". arXiv:2210.03629 — ReAct (10.9).
  • Shinn, N., Cassano, F., Berman, E., et al. (2023). "Reflexion: Language agents with verbal reinforcement learning". arXiv:2303.11366 — Reflexion (10.9).

Chapter 11 — Interpretability and the frontier

Anthropic-Circuits thread (canonical references).

  • Elhage, N., Nanda, N., Olsson, C., et al. (2021). "A mathematical framework for transformer circuits". Transformer Circuits Thread. transformer-circuits.pub — Residual stream, QK/OV decomposition, K/Q/V composition (11.2, 11.3).
  • Olsson, C., Elhage, N., Nanda, N., et al. (2022). "In-context learning and induction heads". Transformer Circuits Thread. transformer-circuits.pub — Induction heads, phase transition (11.4).
  • Elhage, N., Hume, T., Olsson, C., et al. (2022). "Toy models of superposition". Transformer Circuits Thread. transformer-circuits.pub — Superposition theory (11.5).
  • Bricken, T., Templeton, A., Batson, J., et al. (2023). "Towards monosemanticity: Decomposing language models with dictionary learning". Transformer Circuits Thread. transformer-circuits.pub — SAEs for feature extraction (11.6).
  • Templeton, A., Conerly, T., Marcus, J., et al. (2024). "Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet". Transformer Circuits Thread. transformer-circuits.pub — Frontier-scale SAEs; Golden Gate Claude (11.6, 11.7).
  • Anthropic (2024). "Circuit tracing and updates". Transformer Circuits Thread. transformer-circuits.pub — End-to-end circuit attribution (11.9).

Primary papers — circuit analysis.

  • Wang, K., Variengien, A., Conmy, A., et al. (2023). "Interpretability in the wild: A circuit for indirect object identification in GPT-2 small". arXiv:2211.00593 — IOI circuit (11.2, 11.3).
  • Nanda, N., Chan, L., Lieberum, T., et al. (2023). "Progress measures for grokking via mechanistic interpretability". arXiv:2301.05217 — Fourier-basis modular-arithmetic circuit (11.8).
  • Power, A., Burda, Y., Edwards, H., et al. (2022). "Grokking: Generalization beyond overfitting on small algorithmic datasets". arXiv:2201.02177 — Grokking phenomenon (11.8).

Primary papers — sparse autoencoders and steering.

  • Gao, L., Dupré la Tour, T., Tillman, H., et al. (2024). "Scaling and evaluating sparse autoencoders". arXiv:2406.04093 — Top-kk SAEs (11.6).
  • Rajamanoharan, S., Conmy, A., Smith, L., et al. (2024). "Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders". arXiv:2407.14435 — JumpReLU SAEs (11.6).
  • Marks, S., Rager, C., Michaud, E. J., et al. (2024). "Sparse feature circuits: Discovering and editing interpretable causal graphs in language models". arXiv:2403.19647 — Cross-layer SAE circuit identification (11.9).
  • Turner, A., Thiergart, L., Udell, D., et al. (2023). "Activation addition: Steering language models without optimization". arXiv:2308.10248 — Activation steering (11.7).
  • Rimsky, N., Gabrieli, N., Schulz, J., et al. (2023). "Steering Llama 2 via contrastive activation addition". arXiv:2312.06681 — CAA / contrastive steering (11.7). Note: the first author has since published under the name Panickssery, N.
  • Park, K., Choe, Y. J., & Veitch, V. (2023). "The linear representation hypothesis and the geometry of large language models". arXiv:2311.03658 — Linear-representation hypothesis (11.2, 11.5).

Primary papers — emergent abilities debate.

  • Wei, J., Tay, Y., Bommasani, R., et al. (2022). "Emergent abilities of large language models". arXiv:2206.07682 — Emergence claim (11.8).
  • Schaeffer, R., Miranda, B., & Koyejo, S. (2023). "Are emergent abilities of large language models a mirage?". arXiv:2304.15004 — Metric-artefact critique (11.8).
  • Lu, S., Bigoulaeva, I., Sachdeva, R., et al. (2023). "Are emergent abilities in large language models just in-context learning?". arXiv:2309.01809 — Re-analysis of BIG-Bench emergence (11.8).

Primary papers — frontier (model organisms, agents, architectures).

  • Hubinger, E., Denison, C., Mu, J., et al. (2024). "Sleeper agents: Training deceptive LLMs that persist through safety training". arXiv:2401.05566 — Model organisms of deceptive alignment (11.9).
  • van der Weij, T., Hofstätter, F., Jaffe, O., et al. (2024). "AI sandbagging: Language models can strategically underperform on evaluations". arXiv:2406.07358 — Sandbagging studies (11.9).
  • AI21 Labs (2024). "Jamba: A hybrid Transformer-Mamba language model". arXiv:2403.19887 — Jamba hybrid architecture (11.9).
  • Beck, M., Pöppel, K., Spanring, M., et al. (2024). "xLSTM: Extended long short-term memory". arXiv:2405.04517 — xLSTM (11.9).
  • Yang, S., Wang, B., Zhang, Y., et al. (2024). "Parallelizing linear transformers with the delta rule over sequence length". arXiv:2406.06484 — Gated DeltaNet (11.9).
  • Sun, Y., Li, X., Dalal, K., et al. (2024). "Learning to (learn at test time): RNNs with expressive hidden states". arXiv:2407.04620 — Test-Time Training as sequence modelling (11.9).
  • Jimenez, C. E., Yang, J., Wettig, A., et al. (2024). "SWE-bench: Can language models resolve real-world GitHub issues?". arXiv:2310.06770 — SWE-bench (11.9).
  • Xie, T., Zhang, D., Chen, J., et al. (2024). "OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments". arXiv:2404.07972 — OSWorld (11.9).
  • Mialon, G., Fourrier, C., Swift, C., et al. (2023). "GAIA: A benchmark for general AI assistants". arXiv:2311.12983 — GAIA (11.9).
  • Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., & Ha, D. (2024). "The AI Scientist: Towards fully automated open-ended scientific discovery". arXiv:2408.06292 — AI-driven research (11.9). Note: the two "C. Lu" authors are distinct people — Chris Lu and Cong Lu.

Online tools and resources.

  • Anthropic. Transformer Circuits Thread. Living publication for Anthropic's mechanistic-interpretability research.
  • Nanda, N. TransformerLens. PyTorch library for mechanistic interpretability research on small/medium Transformers.
  • EleutherAI. SAE-Lens. Open-source SAE training and analysis toolkit.

Chapter 12 — Capstone: build and train a tiny GPT

The capstone primarily re-uses citations from Chapters 0–11. Specific to this chapter:

For the architectural upgrades in 12.4, see the RoPE and RMSNorm references already in Chapter 6 (Su et al., 2021 — RoPE) and Chapter 3 (Zhang & Sennrich, 2019 — RMSNorm). For the mini-RLHF loop in 12.5, see the DPO and Bradley-Terry references in Chapter 9 (Rafailov et al., 2023; Bradley & Terry, 1952).

Citation conventions used in this course

  • Inline citations use author–year format, e.g. (Vaswani et al., 2017).
  • arXiv references are linked directly to the abstract page.
  • Books are cited by author, year, title, and edition; chapter references where relevant.
  • Lecture series include the lecturer, course name, and a stable URL.
  • Anthropic / OpenAI / Google AI blog posts are cited with author (or "Anthropic", etc.), title, year, and URL.

This bibliography is the source of truth — if a session body cites a paper inline, that paper is listed here. If you find a citation that's missing from this page, file a fix.