The course

From text to a tiny language model

Each localized chapter adds one tested idea to the same Rust implementation.

A map of a modern LLM

See how tokenization, embeddings, decoder blocks, attention, feed-forward layers, training, sampling, and caching fit together in a decoder-only LLM.
Content revision: 5
Text units and vocabulary IDs

Compare UTF-8 bytes, Unicode scalar values, and a demo-only scalar vocabulary before byte-level BPE replaces it.
Content revision: 6
Corpus documents and frozen partitions

Freeze whole source documents into disjoint training, validation, and test partitions before a tokenizer or model can learn from them.
Content revision: 9
Learning deterministic BPE merges

Learn an ordered byte-pair merge table from training documents with explicit rules for overlap, replacement, ties, and document boundaries.
Content revision: 7
Applying and reversing a BPE tokenizer

Replay frozen byte-pair ranks, reserve document controls, and recover every content byte exactly.
Content revision: 9
Building autoregressive input–target pairs

Build next-token input–target pairs inside one encoded document at a time.
Content revision: 8
From transition counts to a bigram model

Count each adjacent training-token transition once, normalize a row, and distinguish a zero probability from a row that cannot be normalized.
Content revision: 5
From assigned probability to perplexity

Turn the probabilities assigned to observed target tokens into mean negative log-likelihood and perplexity while keeping the target count, document boundaries, and evaluated data splits explicit.
Content revision: 7
From tensor coordinates to one flat buffer

Map language-model matrices and attention tensors onto one flat Rust vector with checked row-major strides and deterministic offsets.
Content revision: 5
Shared views and explicit tensor copies

Follow fixed-context word features into Q/K/V tensors and split attention heads, then compare shared tensor views with explicit copies in the course implementation.
Content revision: 6
Align compatible shapes, reduce a named axis

Align feature-wise values across token states, then compute checked sum, mean, and maximum reductions over explicit tensor axes.
Content revision: 5
Multiply rows by columns, then reuse batches

Multiply checked 2-D and batched tensors with scalar Rust loops, including inner-dimension checks, batch broadcasting, and transpose flags.
Content revision: 5
Turn extreme logits into stable probabilities

Turn vocabulary and attention logits into stable probabilities, log-probabilities, log-sum-exp values, and indexed mean NLL with dependency-free Rust.
Content revision: 8
Check gradients before trusting backpropagation

Cross-check selected LLM-training derivatives from actual floating-point probes, with a local-smoothness requirement, scale-aware error, and deterministic tensor coordinates in Rust.
Content revision: 6
Accumulate gradients through a scalar graph

Build reverse-mode scalar autodiff in Rust, accumulate gradients across reused graph edges, and verify them for LLM training.
Content revision: 6
Reverse tensor operations with edge-local VJPs

Build a Rust tensor autodiff tape, reverse shape transformations, broadcasts, and reductions with edge-local VJPs, and verify gradients for LLM training.
Content revision: 9
Reverse the operations that turn token IDs into loss

Implement VJPs for matrix products, repeated embedding lookups, SiLU, log-softmax, and mean token loss, then compare each new local rule with sampled central differences.
Content revision: 7
Initialize trainable weights reproducibly

Initialize model weight matrices reproducibly, compare zero, oversized, and Xavier scales, and track expected variance through stacked linear layers.
Content revision: 4
Give token IDs trainable vectors

Build a trainable token table, validate public token IDs once, pass owned selectors through a private validated gather plan, and scatter-add repeated-token gradients.
Content revision: 7
Mix each token's features with one learned projection

Build a trainable linear layer in Rust, preserve leading token axes, compare affine and bias-free projections, and verify exact reverse gradients.
Content revision: 5
Let one learned branch gate another

Build a position-wise SwiGLU feed-forward layer, follow its activated gate and linear up branches, and verify exact outputs and gradients.
Content revision: 3
Count only the tokens that are really in the batch

Shuffle complete causal windows into mini-batches of fixed-length rows, keep the smaller final batch, and average loss and gradients over its actual target tokens.
Content revision: 3
Keep decay out of the gradient moments

Build AdamW from named parameter gradients, bias-corrected moments, and a separate weight-decay path, then commit every checked update together.
Content revision: 7
Train a fixed-context neural language model

Assemble embeddings, a SwiGLU hidden layer, indexed next-token loss, mini-batches, and AdamW into a deterministic neural n-gram whose held-out loss improves.
Content revision: 4
Keep an identity path around each learned update

Trace exact-shape residual addition, its identity and learned gradient paths, zero-branch learning, and repeated plain versus residual transformations.
Content revision: 3
Normalize scale without subtracting the mean

Implement last-axis RMSNorm, trace its input and gain gradients, and separate ideal scale invariance from epsilon-dominated behavior near zero.
Content revision: 5
Create query, key, and value views

Learn how Transformer self-attention creates query, key, and value tensors from one hidden-state sequence through three independent bias-free projections.
Content revision: 2
Compute one unmasked self-attention head

Learn how one unmasked Transformer self-attention head scores queries against keys, normalizes each row, and mixes values with inspectable Rust evidence.
Content revision: 2
Block future keys with a causal mask

Learn how an inclusive lower-triangular causal mask blocks future Transformer keys, assigns them exactly zero attention probability, and preserves earlier outputs.
Content revision: 2
Turn query and key pairs with RoPE

Learn how rotary position embeddings turn query and key feature pairs by absolute position so relative offsets appear in attention dot products, with a tested Rust implementation.
Content revision: 3
Run causal attention in several heads, then mix

Learn how full-width query, key, and value projections become separate rotary causal attention heads before concatenation and one learned output projection.
Content revision: 2
Compose one pre-norm Transformer decoder block

Learn how RMSNorm, causal multi-head attention, SwiGLU, and two residual paths compose one shape-preserving Transformer decoder block.
Content revision: 2
Stack a decoder and tie its vocabulary head

Learn how token embeddings, repeated causal blocks, final RMSNorm, and one tied vocabulary table produce differentiable next-token logits.
Content revision: 4
Train every step, select with validation

Learn how a decoder training loop orders backpropagation, gradient clipping, scheduled AdamW updates, graph-free validation, and checkpoint selection without using test data.
Content revision: 10
Open one local test gate, keep the report

Learn how one local final-evaluation gate isolates an already selected state, compare graph-free decoder and bigram scores fairly, and distinguish a deliberately selected fixed regression fixture from independent generalization evidence.
Content revision: 7
Save decoder state, replay one specified update

Learn how a versioned checkpoint stores the tokenizer, decoder, trainer-paired AdamW state, and a separate sampling RNG, rejects corrupted bytes, and matches one update whose inputs, targets, and learning rate the caller supplies.
Content revision: 5
Shape the choices, then draw once

Learn how positive temperature, stable top-k filtering, and a restored random-generator state turn decoder logits into controlled, replayable uncached LLM generation.
Content revision: 5
Keep the prefix, project only the new row

Learn how one layer-bound KV cache appends rotated keys and unrotated values while reproducing full-prefix attention at the newest position.
Content revision: 5
Prefill once, then advance one token

Learn how one KV cache per decoder block and one checked model/cache session support prompt prefill followed by coherent one-token decoding, then compare newest-position logits and generation decisions with complete-prefix references.
Content revision: 6
Run the whole tiny LLM

Trace a tiny decoder-only language model in Rust through validation-selected training, a fixed-fixture comparison over overlapping window-target slots, exact reload, and KV-cached generation. Distinguish that comparison from the unreported policy that would score 442 within-document transitions once each with the longest causal prefix capped at four tokens and only its newest-position distribution; numeric NLL and PPL are not reported for that policy.
Content revision: 9