A map of a modern LLM

See the system before its mechanisms

A decoder-only LLM is easier to approach as a connected system than as one opaque box. Prompt text first becomes token IDs. Embedding lookup turns those IDs into learned features. A stack of decoder blocks repeatedly mixes information from the allowed prefix and transforms the features at each position. A vocabulary head then produces one score for every possible next token.

From those scores, the system can do two different jobs. During generation, a decoding policy chooses a token ID, appends it to the sequence, and runs the model again using the existing attention cache. During learning, the observed next token acts as a target: loss measures the prediction, gradients assign responsibility, and an optimizer changes the weights used by embeddings, decoder blocks, and the vocabulary head.

This overview deliberately stops at names, purposes, and connections. Later chapters derive the mathematics and build each mechanism in Rust when that detail has a clear place in the model.

A short road to the modern block diagram

Count n-grams condition on a fixed short context and cannot learn reusable distributed features or content-dependent access to a longer prefix. Neural language models learned distributed token representations, the Transformer supplied masked self-attention over the prefix, and later autoregressive Transformer language models scaled next-token prediction. A modern decoder-only LLM repeatedly transforms token features with normalized causal attention and gated feed-forward branches, then projects the result to a next-token distribution used by learning or generation.

A Neural Probabilistic Language Model supports the earlier boundary. Bengio and colleagues contrast n-gram generalization with a neural probability function that learns distributed word representations; their model predates the Transformer and this course’s decoder block.

Attention Is All You Need supports the later architecture boundary. Vaswani and colleagues define the Transformer and mask decoder self-attention against later positions; their published system is an encoder-decoder model, not this course’s decoder-only topology.

Language Models are Few-Shot Learners supports the scaling boundary. Brown and colleagues describe GPT-3 as a scaled autoregressive language model; its scale and measured capabilities do not transfer to the tiny reference model used here.

Follow the blocks as one connected system

The first figure is the whole-system schema: a shared forward path reaches logits, then generation and learning branch in different directions. Feedback arrows show where a chosen token re-enters the model and where updated weights affect the next learning step. The cache and numeric foundation sit beside the path because they support model computation rather than acting as extra tokens in the sequence.

The second figure keeps the useful detailed views. Its three panels list the ordinary inference stages, open the repeated decoder block, and separate the post-logit learning operations. Every named part links to the chapter that implements it.

How the complete system connects

The forward path produces logits. Generation chooses another input token; learning changes the weights that produced those logits.

Shared forward path

Text / examples

Prompt input; separate training documents.

→

Tokenizer

Text to token IDs and back.

→

Embeddings

Token IDs to learned features.

→

Decoder stack

Repeat attention and gated feedforward layers.

↻ repeat for every layer

→

Vocabulary head

Final features to token logits.

→

Logits

One score for every vocabulary token.

↙ Generation branch

Sampler

Turn logits into probabilities and choose a token.

→

Chosen token ID

Append it to the sequence.

↺ The chosen ID returns to embedding lookup for the next cached step.

↘ Learning branch

Forward logits

Model scores for this position.

and

Target token

Observed token ID.

→

Loss

Compare the logits with the observed token.

→

Gradients

Show how each weight affected the loss.

→

AdamW optimizer

Use gradients to change the weights.

→

Updated model weights

Validation later selects one saved state.

↺ Updated weights feed the shared forward path on the next training step.

State and foundations attached to the path

Per-layer KV cache

Decoder attention reads and extends this cache during generation.

Shared numeric core

Tensor operations run every block; autodiff records learning.

Evaluation

Selected weights → one local evaluator → fixed-fixture report.

Checkpoint

Model and training state ↔ saved checkpoint.

The next-token inference path

Tokenize the prompt once; each chosen token ID loops back to embeddings for the next cached decode.

1. 1 Text / examples

   Prompt input; separate training documents.

   • Ch 02 ,
   • Ch 05 ,
   • Ch 21

2. 2 Tokenizer

   Text to token IDs and back.

   • Ch 01 ,
   • Ch 03 ,
   • Ch 04

3. 3 Embeddings

   Token IDs to learned features.

   • Ch 18

4. 4 Decoder stack

   Repeat attention and gated feedforward layers.

   ↻ repeat for every layer

   • Ch 31

5. 5 Vocabulary head

   Final features to token logits.

   • Ch 32

6. 6 Sampler

   Turn logits into probabilities and choose a token.

   ↺ token ID to embeddings

   • Ch 12 ,
   • Ch 36

Inside every pre-norm decoder block

Each block adds attention, then a feed-forward update.

1. 1 RMSNorm

   Set branch input scale.

   • Ch 25

2. 2 Causal attention

   Read the allowed prefix.

   • Ch 26 ,
   • Ch 27 ,
   • Ch 28 ,
   • Ch 29 ,
   • Ch 30

3. 3 Residual add

   Add a branch update to the stream.

   • Ch 24

4. 4 RMSNorm

   Set branch input scale.

   • Ch 25

5. 5 SwiGLU

   Gated transform at each position.

   • Ch 20

6. 6 Residual add

   Add a branch update to the stream.

   • Ch 24

Learning and evaluating the same weights

Use target loss to update the same weights.

↘ Reuse the forward path through logits

text → tokenizer → embeddings → decoder blocks → vocabulary head; the observed target replaces sampling.

6. 6 Loss

   Compare the logits with the observed token.

   • Ch 06 ,
   • Ch 07 ,
   • Ch 23

7. 7 AdamW and selection

   AdamW updates weights; validation chooses a state.

   • Ch 22 ,
   • Ch 33

8. 8 Evaluation

   Score the selected frozen model after selection; retain the known result only as fixed-fixture regression evidence.

   • Ch 34

9. 9 Checkpoint

   Save the exact model and training state.

   • Ch 35

Use the map as a table of contents

You do not need to memorize this map. Return to it whenever a later mechanism feels isolated and ask only where that mechanism sits: before the decoder, inside each repeated block, after logits, or in the process that trains and preserves the weights.

Chapters 1 through 7 establish text, tokenization, causal examples, a count baseline, and language-model metrics. Chapters 8 through 23 build the numeric and learning foundation. Chapters 24 through 32 assemble the decoder. Chapters 33 through 38 train, evaluate, persist, sample, and cache it. Chapter 39 runs the whole tiny LLM.

The evaluation node names a post-selection role, not a promise that every repository run sees newly unopened data. Inside one execution, the local evaluator cannot affect the selected state; repository reruns retain the known result only as fixed-fixture regression evidence.

Start with the model’s input boundary

Chapter 1 begins the implementation path with the distinction every later block depends on: text is not yet a model input. It must first become stable token IDs. From there, each chapter will replace one label in this overview with an explanation, executable Rust, and evidence that the part works.