Run the whole tiny LLM

Predict the boundary before predicting the output

The corpus contains $8$ training documents, $2$ validation documents, and $2$ test documents. The first prediction is about information flow: only the training documents may teach BPE merge ranks, update decoder parameters, or fit the count baseline. Validation may choose a trained state. Test evidence may do neither.

Eight training-only BPE merges produce vocabulary size $266$. With decoder context capacity $C=4$ and stride one, the encoded partitions produce $1820$, $463$, and $436$ overlapping causal windows. Updates use mini-batches of $16$ windows; the three evaluation epochs use mini-batches of at most $128$ windows, so they contain $15$, $4$, and $4$ mini-batches. The decoder has one block, one head, width $4$, feed-forward width $4$, and $1188$ learned scalar parameters.

Before reading the result, predict:

1. whether two runs from training seed $39$ select the same state bit for bit;
2. whether test loss is visible before that state is fixed;
3. whether checkpoint reload changes any logit for the frozen At probe; and
4. whether cached generation changes a sampling decision.

Both training replays select step $32$ at validation loss $3.889531885$. Only after selection does the run materialize test mini-batches and give them to one local final evaluator. Every one of the $436$ overlapping test windows contributes $C=4$ target slots:

A window-target slot is identified by its document, window start, and position inside the window. Because adjacent windows overlap, one within-document transition occurrence can enter the score in as many as four slots. The decoder sees one, two, three, or four in-window context tokens at the four causal slot positions; $C=4$ is its capacity, not the context length of every prediction. The bigram always uses only the immediately preceding token.

Across the two documents, $4$ transition occurrences appear in one slot, $4$ in two slots, $4$ in three slots, and $430$ in four slots. The multiplicities recover the reported denominator:

The decoder and frozen alpha-one bigram score the same ordered $1744$ slots, including those repetitions. Their reported values are mean NLL in nats per slot: $3.866087547$ for the decoder and $3.981342714$ for the bigram. The window-slot perplexity is the exponential of the corresponding mean:

The exact window-slot perplexities are $47.755180205$ for the decoder and $53.588940583$ for the bigram. Perplexity is dimensionless.

Thus $3.866087547$ is mean NLL, not perplexity. The mean-NLL gap is $0.115255167$ on this fixed fixture. Scoring the same ordered slots makes that within-fixture comparison fair for this slot-weighted protocol.

The same two test documents contain $444$ encoded tokens and therefore $442$ within-document transition occurrences:

A conventional once-per-transition metric would score each occurrence once. It would present the decoder with the maximal available prefix of at most four immediately preceding tokens and use only the distribution at the newest position. Chapter 39 does not report that metric’s mean NLL or perplexity, so the $1744$-slot values must not be read as corpus scores over $442$ transitions.

The lower decoder slot mean-NLL ordering is retained across later executions, so it is useful regression evidence. It is not an untouched independent estimate of generalization or evidence of architecture-wide decoder superiority.

The $30994$-byte checkpoint re-encodes to the same bytes and restores the model, optimizer, tokenizer, and random state exactly. A separate probe At, encoded as [67,118], reproduces every logit bit. Generation then starts from prompt A, encoded as literal token ID 67. Sampling with $\tau=0.8$, $k=4$, and seed $38$ chooses literal IDs [260,34,34], decoded as т␠␠, where each ␠ marks one generated space. Cached and complete-prefix paths use the same draws and make the same decisions.

One product connects every next-token decision

The complete model still answers the same question introduced near the start of the course:

The sequence probability $P_\theta(z_{1:T})$ is a product of conditional probabilities. Training minimizes the negative log of those conditionals; validation selects $\theta$; final evaluation measures the already selected $\theta$; and generation samples one new $z_t$ from the conditional distribution given $z_{<t}$.

The factorization is a causal promise. The probability for position $t$ may depend on $z_{<t}$, never on a later token. Here $z_{<t}$ names the whole earlier prefix in the general factorization; this bounded decoder actually receives at most the retained last $C=4$ earlier tokens. A frozen split makes the experimental promise parallel to the mathematical one: test-reserved documents cannot affect tokenizer learning, updates, or validation selection in this run. That within-execution boundary does not make a known result independently held out again when a later execution reuses it.

The test comparison instantiates that factorization with an explicit observation list. If slot $i$ contains observed target $z_i$ and the context actually visible at that slot is $c_i$, then the reported mean is

This sum has $N_{\mathrm{slot}}=1744$ equally weighted terms. Equal slot weight does not mean equal transition weight, because overlap repeats a within-document transition occurrence. The separate $442$-term policy would score each occurrence once, use the longest available causal prefix capped at four tokens and only its newest-position distribution, and report no numeric mean NLL or PPL here.

Keep sequence position separate from pipeline stage

• $P_\theta$ is the probability distribution defined by the decoder.
• $\theta$ contains all learned parameter values chosen by validation loss.
• $z_{1:T}$ is one sequence from its first token through token $T$.
• $T$ is the number of tokens whose joint probability is being assigned.
• $\prod_{t=1}^{T}$ multiplies one conditional term at every position.
• $t$ is the current token position.
• $z_t$ is the observed token at that position.
• $z_{<t}$ is the earlier causal prefix in the general factorization; this fixture presents at most the last $C=4$ earlier tokens to the decoder.
• $N_{\mathrm{slot}}$ is the number of overlapping window-target slots scored by each model; here it is $1744$.
• $\mathcal L_{\mathrm{slot}}$ is mean NLL in nats per slot, and $\operatorname{PPL}_{\mathrm{slot}}$ is its dimensionless exponential.
• $N_{\mathrm{transition}}$ counts $442$ within-document transition occurrences. The separate policy would score each once with the longest available causal prefix capped at four tokens and only its newest-position distribution; its numeric mean NLL and PPL are not reported.

The training step number is not $t$. A step updates $\theta$ from a mini-batch; $t$ indexes a token inside the language-model factorization.

From short count contexts to autoregressive Transformer LLMs

A count-based bigram estimates the next token from one preceding token and cannot share statistical strength through learned features or use the longer causal prefix.

A Neural Probabilistic Language Model supports this bounded claim: Bengio and colleagues describe traditional n-gram generalization through short overlapping sequences and show a neural probability function that learns distributed word representations and benefits from longer contexts; their model is not a Transformer or this course pipeline.

Generalization in Adaptive Data Analysis and Holdout Reuse supplies the evidence warning. Dwork and colleagues show that adaptive repeated reuse of a standard holdout can overfit that holdout; this general warning does not establish any fact about the capstone fixture, its score, or its local access count.

Attention Is All You Need supports a later boundary: Vaswani and colleagues define the Transformer and mask decoder self-attention so a position cannot read later positions; their published architecture is an encoder-decoder model and does not define this course data, checkpoint, or generation policy.

Language Models are Few-Shot Learners supplies the scaled-LLM context: Brown and colleagues train GPT-3, a 175-billion-parameter autoregressive Transformer language model based on the GPT-2 architecture, and evaluate zero-, one-, and few-shot tasks without gradient updates or fine-tuning; none of its scale or capability results transfers to this tiny teaching run.

Neural language models learned distributed token features and longer-context probability functions; the Transformer supplied masked self-attention, and later autoregressive Transformer language models scaled that training objective.

This course capstone combines training-only tokenizer learning, causal next-token updates, validation-selected state, within-execution selection isolation, fixed-fixture regression evaluation, exact checkpoint round-trip, and stateful generation; these are local evidence rules, not requirements of the cited papers.

Compare the training-only alpha-one bigram with the validation-selected causal decoder on the same ordered overlapping window-target slots. Four tokens is the decoder’s context capacity; the four slot positions expose one through four in-window context tokens. The mean-NLL gap retained by later executions is fixed-fixture regression evidence, not causal attribution to context or attention and not an independent generalization estimate. Bengio and colleagues’ discussion of overlapping sequences does not define this course’s stride-one slot weighting.

Count n-grams provided a strong short-context baseline; learned distributed features and masked self-attention made longer learned computation possible, and scaled autoregressive Transformers became one major family of modern LLMs. This capstone demonstrates local end-to-end responsibility boundaries at inspectable scale while treating the known lower decoder window-slot mean NLL only as a regression condition for this fixed fixture.

The executable contrast is narrow: a one-token count context and a decoder with four-token context capacity score the same $1744$ repeated window slots on this frozen corpus. The lower slot mean NLL verifies the retained fixed-fixture ordering. The separate policy would score the $442$ transition occurrences once each with the longest causal prefix capped at four tokens and only its newest-position distribution, but its numeric mean NLL and PPL are not reported. The ordering is also not an untouched independent estimate of generalization, a universal architecture ranking, or evidence of useful generation quality.

#[derive(Clone, Debug, PartialEq)]
pub struct HistoricalContrast {
    pub window_slot_unit: &'static str,
    pub window_target_slots: usize,
    pub document_transition_occurrences: usize,
    pub bigram_context_tokens: usize,
    pub decoder_context_capacity: usize,
    pub decoder_window_slot_context_lengths: Vec<usize>,
    pub bigram_window_slot_mean_nll_nats: f64,
    pub decoder_window_slot_mean_nll_nats: f64,
    pub window_slot_gap_nats: f64,
}

/// Measures the fixture's short-context baseline against its causal decoder.
pub fn historical_contrast(evidence: &CapstoneRun) -> HistoricalContrast {
    let evaluation = evidence.final_evaluation();
    let decoder_context_capacity = evidence.training().model_config().max_positions();
    HistoricalContrast {
        window_slot_unit: WINDOW_SLOT_UNIT,
        window_target_slots: evaluation.window_target_slot_count(),
        document_transition_occurrences: evaluation.document_transition_occurrence_count(),
        bigram_context_tokens: 1,
        decoder_context_capacity,
        decoder_window_slot_context_lengths: (1..=decoder_context_capacity).collect(),
        bigram_window_slot_mean_nll_nats: evaluation.bigram().mean_nll(),
        decoder_window_slot_mean_nll_nats: evaluation.decoder().mean_nll(),
        window_slot_gap_nats: evaluation.loss_gap(),
    }
}

Assemble APIs instead of copying algorithms

run_capstone starts by parsing the checked corpus and split. It learns BPE from the training partition, encodes each document without joining boundaries, fits the alpha-one bigram from the same training tokens, and builds separate causal epochs for training updates and validation. It does not build the test epoch at this stage.

Two decoder runs receive the same initialization and batch-order seed. Each executes all $32$ updates. The implementation compares every training event, loss checkpoint, optimizer moment, selected step, and model value by bit pattern. Only after replay agrees and validation fixes the selection does the run prepare final evaluation. TrainingResult owns both the retained selected state and an independent decoder with the same values. SelectedDecoder borrows both. The run then materializes the test epoch, records two facts—$436$ windows and $4$ mini-batches—and moves the epoch into FinalEvaluator. The evaluator owns the epoch and the local permission to evaluate it once. Immediately before opening that local test gate, it verifies that the decoder’s configuration, ordered names and shapes, and every parameter bit still match the retained state; this prevents a changed decoder from being evaluated under stale selection metadata without making another model copy. The evaluator then scores the borrowed model and consumes its permission. It compares both models on the same ordered $1744$ overlapping window-target slots, including the same repeated transition occurrences. The decoder has context capacity four, but its slot predictions use actual in-window context lengths $1$, $2$, $3$, and $4$. The separate policy would score all $442$ occurrences once each, use the longest available causal prefix capped at four tokens and only its newest-position distribution, and report no numeric mean NLL or PPL.

The permission, not either borrowed object, is the one-use resource. This is an executable boundary inside this run, not a claim that the repository globally prevents other code from reading the corpus. Later report assembly retains the derived counts and metric evidence, not a cloned epoch.

The pipeline retains the lower decoder window-slot mean NLL as an explicitly named fixed-fixture regression condition; FinalEvaluator itself remains neutral to which model is lower. The learner evidence uses decoder_lower_on_fixture:true and records scope:fixed-fixture-regression, within_run_selection_isolated:true, independent_generalization_estimate:false, and architecture_superiority_evidence:false.

The report derives the displayed count of $1188$ learned scalars by calling primary.selected_state().scalar_count(). That method computes the count by summing the lengths of the parameter tensors in the retained graph-free state selected by validation; it neither reads a stored count nor initializes another decoder.

Final evaluation and durable checkpoint storage have different ownership needs. Evaluation only borrows the retained selected state and matching decoder. After evaluation, the checkpoint must coexist with the complete training result, so Checkpoint::from_snapshot deliberately copies the selected graph-free state and optimizer persistence state. That copy is required because the training result and checkpoint must remain independently usable.

Re-encoding the loaded checkpoint must reproduce the saved bytes, and its model and optimizer bits, BPE ranks, step, and RNG state must all agree. Before consuming the checkpoint, the pipeline records the tokenizer, selected step, optimizer state, and saved RNG value needed by later evidence. It then calls loaded.into_model(), which moves the owned model buffers into a decoder. The separate At probe compares that decoder with primary.selected_model() and must match bit for bit. Only then does the loaded model run cached and complete-prefix generation from prompt A and the saved RNG state. generation_evidence encodes the prompt once into one prompt_ids vector. Both generation paths temporarily read that vector through immutable references. After both calls finish and the local vector is no longer needed for calculation, GenerationEvidence takes ownership of the same vector; moving the Vec transfers its existing buffer instead of cloning the prompt IDs for the report.

pub fn run_capstone(
    corpus_source: &str,
    split_source: &str,
    checkpoint_path: impl AsRef<Path>,
    config: CapstoneConfig,
) -> Result<CapstoneRun, PipelineError> {
    let data = prepare_data(corpus_source, split_source, config)?;
    let prepared = prepare_training(&data, config)?;
    let primary = training_once(&prepared, config)?;
    let replay = training_once(&prepared, config)?;
    let replay_bitwise = training_replays_bitwise(&primary, &replay);
    require(replay_bitwise, "same-seed training replay changed bits")?;
    let selected_step = primary.selected_step();
    let selected_step_u64 = u64::try_from(selected_step)
        .map_err(|_| PipelineError::invariant("selected step does not fit u64"))?;
    require(
        selected_step == config.updates(),
        "validation no longer selects the final optimizer state",
    )?;
    require(
        selected_step_u64 == primary.selected_optimizer_state().step_count(),
        "selected model and optimizer no longer share one step",
    )?;
    require(
        primary.selected_state().bit_pattern() == primary.final_state().bit_pattern(),
        "selected and final model states diverged",
    )?;

    let provenance = map(
        PipelineStage::FinalEvaluation,
        EvaluationProvenance::new(
            data.partitions.corpus_checksum(),
            format!(
                "{}:{}",
                data.partitions.split_strategy(),
                data.partitions.corpus_checksum()
            ),
            format!(
                "byte-bpe-v{}-merges{}",
                TOKENIZER_LAYOUT_VERSION,
                data.tokenizer_evidence.learned_merges()
            ),
            config.context_length(),
        ),
    )?;
    let decoder = map(
        PipelineStage::FinalEvaluation,
        SelectedDecoder::new(
            primary.selected_state(),
            primary.selected_model(),
            primary.selected_step(),
            primary.selected_validation_loss(),
            Partition::Validation,
            &provenance,
        ),
    )?;
    let frozen_bigram = map(
        PipelineStage::FinalEvaluation,
        FrozenBigram::new(&data.bigram, Partition::Train, &provenance),
    )?;
    let test_epoch = epoch(
        &data.encoded,
        Partition::Test,
        config.context_length(),
        config.window_stride(),
        config.evaluation_batch_size(),
        BatchOrder::Sequential,
    )?;
    require(
        config.window_stride() == 1
            && epoch_matches_window_stride(&test_epoch, config.window_stride()),
        "the Chapter 39 evaluation must retain complete stride-one windows",
    )?;
    let test_window_count = test_epoch.window_count();
    let test_batch_count = test_epoch.batch_count();
    let mut evaluator = map(
        PipelineStage::FinalEvaluation,
        FinalEvaluator::new(test_epoch, provenance.clone()),
    )?;
    require(
        evaluator.access_count() == 0,
        "test evidence opened before model selection completed",
    )?;
    let final_evaluation = map(
        PipelineStage::FinalEvaluation,
        evaluator.evaluate_once(decoder, frozen_bigram),
    )?;
    let expected_window_target_slots = test_window_count
        .checked_mul(config.context_length())
        .ok_or_else(|| PipelineError::invariant("test window-target slot count overflowed"))?;
    let expected_document_transition_occurrences = data
        .encoded
        .documents(Partition::Test)
        .iter()
        .map(|document| document.token_ids().len().saturating_sub(1))
        .sum::<usize>();
    let multiplicity_slot_count = final_evaluation
        .transition_multiplicity_counts()
        .iter()
        .enumerate()
        .map(|(index, count)| (index + 1) * count)
        .sum::<usize>();
    let multiplicity_transition_count = final_evaluation
        .transition_multiplicity_counts()
        .iter()
        .sum::<usize>();
    require(
        final_evaluation.window_target_slot_count() == expected_window_target_slots
            && final_evaluation.document_transition_occurrence_count()
                == expected_document_transition_occurrences
            && multiplicity_slot_count == expected_window_target_slots
            && multiplicity_transition_count == expected_document_transition_occurrences,
        "test window slots, document transitions, and overlap multiplicities disagree",
    )?;
    require(
        final_evaluation.decoder_has_lower_loss(),
        "fixed capstone fixture no longer records lower decoder loss than its frozen bigram",
    )?;

    let rng_state = SplitMix64::from_seed(config.generation_seed()).state();
    let checkpoint = map(
        PipelineStage::Checkpoint,
        Checkpoint::from_snapshot(
            CheckpointTokenizer::byte_bpe(&data.tokenizer),
            primary.selected_training_state(),
            rng_state,
        ),
    )?;
    let encoded_checkpoint = map(
        PipelineStage::Checkpoint,
        checkpoint.save_atomic(checkpoint_path.as_ref()),
    )?;
    let loaded = map(
        PipelineStage::Checkpoint,
        Checkpoint::load(checkpoint_path.as_ref()),
    )?;
    let loaded_encoded = map(PipelineStage::Checkpoint, loaded.encode())?;
    let bytes_roundtrip = encoded_checkpoint.bytes() == loaded_encoded.bytes();
    require(
        bytes_roundtrip,
        "loaded checkpoint bytes differ from the saved record",
    )?;
    let loaded_tokenizer = map(PipelineStage::Checkpoint, loaded.tokenizer().restore_bpe())?
        .ok_or_else(|| PipelineError::invariant("checkpoint lost its byte BPE tokenizer"))?;
    let model_bits_exact =
        loaded.model_state().bit_pattern() == primary.selected_state().bit_pattern();
    let optimizer_bits_exact =
        adamw_state_bitwise(loaded.optimizer_state(), primary.selected_optimizer_state());
    let loaded_selected_step = loaded.selected_step();
    let loaded_optimizer_step = loaded.optimizer_state().step_count();
    let loaded_rng_state = loaded.sampling_rng_state();
    let loaded_model = map(PipelineStage::Checkpoint, loaded.into_model())?;
    let logit_probe_text = "At";
    let logit_probe_ids = data.tokenizer.encode_utf8(logit_probe_text);
    let prompt_logits_bitwise = logits_bits(primary.selected_model(), &logit_probe_ids)?
        == logits_bits(&loaded_model, &logit_probe_ids)?;
    let tokenizer_exact = loaded_tokenizer == data.tokenizer;
    require(
        model_bits_exact && optimizer_bits_exact && prompt_logits_bitwise && tokenizer_exact,
        "checkpoint reload changed model, optimizer, tokenizer, or probe logits",
    )?;
    let generation =
        generation_evidence(&loaded_model, &loaded_tokenizer, loaded_rng_state, config)?;

    let checkpoints = primary
        .checkpoints()
        .iter()
        .map(|checkpoint| TrainingCheckpointEvidence {
            step: checkpoint.step(),
            train_loss: checkpoint.train().mean_loss(),
            validation_loss: checkpoint.validation().mean_loss(),
            selected: checkpoint.selected(),
        })
        .collect();
    let training = TrainingEvidence {
        model_config: prepared.model_config,
        parameter_count: primary.selected_state().scalar_count(),
        window_counts: [
            prepared.train.window_count(),
            prepared.validation.window_count(),
            test_window_count,
        ],
        batch_counts: [
            prepared.train.batch_count(),
            prepared.validation.batch_count(),
            test_batch_count,
        ],
        checkpoints,
        selected_step,
        selected_validation_loss: primary.selected_validation_loss(),
        optimizer_step: primary.final_optimizer().step_count(),
        replay_bitwise,
    };
    let checkpoint_evidence = CheckpointEvidence {
        bytes: encoded_checkpoint.bytes().len(),
        header_bytes: encoded_checkpoint.header_bytes(),
        checksum: encoded_checkpoint.checksum_label(),
        tensor_records: encoded_checkpoint.tensors().len(),
        selected_step: loaded_selected_step,
        optimizer_step: loaded_optimizer_step,
        sampling_rng_state: loaded_rng_state,
        bytes_roundtrip,
        model_bits_exact,
        optimizer_bits_exact,
        tokenizer_exact,
        logit_probe_text: logit_probe_text.to_owned(),
        logit_probe_ids,
        prompt_logits_bitwise,
    };
    Ok(CapstoneRun {
        partitions: data.partitions,
        tokenizer: data.tokenizer_evidence,
        training,
        final_evaluation,
        checkpoint: checkpoint_evidence,
        generation,
    })
}

The runnable evidence requires every final claim together. It rejects a changed test ordering, a changed lower-decoder slot-metric regression condition, a replay with different bits, altered checkpoint state or probe logits, or any cached sampling divergence. That requirement protects known fixture behavior; it does not promote the ordering into independent generalization evidence.

/// Runs the checked corpus-to-generated-text program and checks its final claims.
pub fn learner_evidence() -> Result<CapstoneRun, FixtureError> {
    let checkpoint = TemporaryCheckpoint::new();
    let evidence = run_capstone(
        CORPUS_JSON,
        SPLITS,
        checkpoint.path(),
        CapstoneConfig::tiny(),
    )?;
    require(
        evidence.final_evaluation().decoder_has_lower_loss(),
        "fixed capstone fixture no longer records lower decoder loss than its frozen bigram",
    )?;
    require(
        within_run_selection_isolated(&evidence),
        "within-run selection isolation evidence changed",
    )?;
    require(
        evidence.training().replay_bitwise(),
        "same-seed training replay changed bits",
    )?;
    require(
        evidence.checkpoint().bytes_roundtrip()
            && evidence.checkpoint().model_bits_exact()
            && evidence.checkpoint().optimizer_bits_exact()
            && evidence.checkpoint().tokenizer_exact()
            && evidence.checkpoint().prompt_logits_bitwise(),
        "checkpoint reload changed bytes, model, optimizer, tokenizer, or probe logits",
    )?;
    require(
        evidence.generation().tokens_exact()
            && evidence.generation().decisions_bitwise()
            && evidence.generation().rng_state_exact(),
        "cached generation changed reference decisions",
    )?;
    Ok(evidence)
}

Run cargo run —quiet —locked -p ch39-end-to-end-llm. The executable prints this exact report:

chapter=39-end-to-end-llm
data=checksum:fnv1a64:723b071980ae8a22 split:fixed-paired-document-holdout-v1 documents:8/2/2 train_ids:[en-river-dawn,ru-river-dawn,en-clock-shop,ru-clock-shop,en-rain-library,ru-rain-library,en-bee-garden,ru-bee-garden] validation_ids:[en-night-station,ru-night-station] test_ids:[en-winter-window,ru-winter-window]
tokenizer=layout:1 requested:8 learned:8 training_only:true vocabulary:266 encoded_tokens:[1852,471,444]
model=layers:1 heads:1 width:4 feed_forward:4 context:4 parameters:1188 update_batch_size:16 evaluation_batch_size:128 windows:[1820,463,436] evaluation_batches:[15,4,4]
training=updates:32 seed:39 checkpoints:0:5.621745486/5.628342353/candidate;32:3.855502695/3.889531885/selected selected:32 validation:3.889531885 optimizer:32 replay_bitwise:true
test=access:1 documents:[en-winter-window,ru-winter-window] stride:1 windows:436 batches:4 window_target_slots:1744 document_transition_occurrences:442 transition_multiplicity_counts:[1x4,2x4,3x4,4x430] window_slot_fingerprint:fnv1a64:77b836869f848986 no_grad:true unchanged:true
slot_metric=unit:overlapping-window-target-slot decoder_window_slot_mean_nll_nats:3.866087547 decoder_window_slot_perplexity:47.755180205 bigram_window_slot_mean_nll_nats:3.981342714 bigram_window_slot_perplexity:53.588940583 window_slot_gap_nats:0.115255167 comparison_slot_set:shared-ordered-window-slots decoder_lower_on_fixture:true
transition_metric=unit:within-document-next-token-transition count:442 context_policy:longest-available-causal-prefix-up-to-4 newest_position_only:true reported:false mean_nll:not-reported perplexity:not-reported
evidence=scope:fixed-fixture-regression within_run_selection_isolated:true independent_generalization_estimate:false architecture_superiority_evidence:false
checkpoint=bytes:30994 header:2418 records:34 checksum:fnv1a64:67aeaaea603b291f selected:32 optimizer:32 rng:0x0000000000000026 bytes_roundtrip:true model_bits_exact:true optimizer_bits_exact:true tokenizer_exact:true logit_probe:At logit_probe_ids:[67,118] prompt_logits_bitwise:true
generation=prompt:A prompt_ids:[67] temperature:0.8 top_k:4 seed:38 generated:[260,34,34] text:"т  " prefixes:[1,2,3] stop:token-limit prefill:1 decode:2 final_cache:3 cached_scores:6 calculated_complete_prefix_scores:14 rng_initial:0x0000000000000026 rng_final:0xdaa66d2c7ddf7465 tokens_exact:true decisions_bitwise:true rng_exact:true
history=window_slot_unit:overlapping-window-target-slot window_target_slots:1744 document_transition_occurrences:442 bigram_context_tokens:1 decoder_context_capacity:4 decoder_window_slot_context_lengths:[1,2,3,4] bigram_window_slot_mean_nll_nats:3.981342714 decoder_window_slot_mean_nll_nats:3.866087547 window_slot_gap_nats:0.115255167
next=inspect, modify, test, and extend the complete decoder

fn main() -> Result<(), Box<dyn std::error::Error>> {
    print!("{}", ch39_end_to_end_llm::learner_report()?);
    Ok(())
}

Follow the one-way pipeline

The figure begins with fixed document roles and training-only tokenizer learning. It then follows causal windows into the decoder and places the validation-selected state before a double-marked local test gate. Test loss has no path back to an update or selection in this run. The test card identifies the $1744$ observations as overlapping window-target slots and labels the values as mean NLL in nats per slot. The separate policy would score all $442$ within-document transition occurrences once each, use the longest causal prefix capped at four tokens and only its newest-position distribution, and report no numeric mean NLL or PPL. The known ordering remains a fixed-fixture regression condition, not independent generalization evidence.

The last two stages ask different equality questions. Checkpoint reload preserves saved bytes plus model, optimizer, tokenizer, step, and RNG state exactly; logits for probe At agree bit for bit. Cached generation from prompt A preserves the complete-prefix token decisions and RNG state. Before the three token choices, the retained prefixes have lengths $1$, $2$, and $3$ tokens. Cache prefill processes the one-token prompt to produce logits for the first choice; two one-token decode calls then process the first two generated tokens to produce logits for the next two choices. The cached path’s instrumented attention work contains $1+2+3=6$ score cells; the same prefix schedule gives the calculated complete-prefix reference count $1^2+2^2+3^2=14$.

Predict before checking the final trace

1. Which partition teaches the eight BPE merges?
2. How many decoder parameters receive updates?
3. What evidence chooses step $32$?
4. May the test loss change that choice?
5. How do $436$ overlapping test windows become $1744$ window-target slots, and why are there only $442$ within-document transition occurrences?
6. What is the measured slot mean-NLL gap, how is window-slot perplexity related to each model’s slot mean NLL, and what evidence scope does the ordering have?
7. Which checkpoint facts are exact, and which claim belongs only to probe At?
8. Which literal token IDs follow prompt A?
9. Why is decoded т␠␠ not evidence of translation quality?
10. What does the second seed-$39$ training run prove?

Misconception: access:1 means the repository has used this test result only once. The count belongs to one evaluator instance inside one execution. This run freezes the state first and materializes test batches afterward, so test cannot affect that selected state. Later executions reuse the known ordering for regression checking; determinism makes that repeated comparison reproducible, not independent.

Take ownership of the complete decoder

Every course component now participates in one functional program: frozen bilingual data becomes BPE tokens and causal batches; validation selects the decoder before the local final evaluator receives test batches; both models score the same ordered overlapping window-target slots; the separate policy would score $442$ within-document transitions once each with the longest causal prefix capped at four tokens and only its newest-position distribution, but reports no numeric mean NLL or PPL; the known lower decoder slot mean-NLL ordering remains only fixed-fixture regression evidence; checkpoint bytes and state round-trip exactly; the separate At probe reproduces logits bit for bit; and cached generation from A returns decoded text.

That completes the planned road to a modern decoder-only LLM at teaching scale. You can now change one bounded piece—more context, a wider block, another tokenizer budget, a different sampler—and identify exactly which data, mathematical, training, evaluation, persistence, and inference evidence must be re-established.