Compare training and held-out language-model loss

Separate a better training fit from better held-out scores

An optimizer receives the training targets and changes weights to reduce their loss. That feedback can improve the fit to those particular documents while making separate documents less likely. A falling training loss therefore leaves an important part of the result unmeasured. We need to freeze the same learned weights, score both training and held-out targets without updating them, and keep the scoring rules comparable.

Use Chapter 4’s valid-target masks, Chapter 5’s document-local packed contexts, Chapter 8’s training and evaluation modes, Chapter 13’s accumulated updates and Chapter 15’s portable model as prerequisites. We extend their existing model, loss, trainer and saved-state paths.

Follow one measured run

Fix two training documents, aaab and aab, and two validation documents, abaa and baa. Only the training documents enter optimizer updates. For initialization seed $39$, the measured training mean negative log-likelihood (NLL) falls from $0.874321$ nats per target after update $20$ to $0.157051$ after update $80$. Over the same interval, validation NLL rises from $1.267297$ to $4.172745$. Lower training loss has accompanied worse validation likelihood.

The rise is visible in each of the two observed intervals, $20\to40$ and $40\to80$, for all three predeclared seeds $39$, $40$ and $41$. This is the experiment’s bounded overfitting signal. A training/validation gap alone would not establish that change over time. Different string order and difficulty can also affect a gap between the splits.

Keep the target occurrences explicit

The existing Chapter 4 tokenizer is fixed before these examples are encoded; validation is not used to fit it. Its document encoding includes beginning-of- sequence (BOS) ID 0, content IDs 99 for a and 100 for b, and end-of-sequence (EOS) ID 1.

Each document supplies one fewer next-token target than encoded IDs: BOS supplies context, while EOS is an observed target. Each split therefore has $5+4=9$ valid targets. Padding supplies none. Document IDs and complete texts are distinct across the splits, but the strings share symbols and are deliberately related. This is a constructed demonstration, not a random sample of natural language. Validation is inspected repeatedly; no untouched final test set is used or claimed.

Add token losses before taking a mean

A short batch must not receive the same total weight as a longer batch merely because both produce one mean. To expose the denominator, use two constructed lists of assigned target probabilities: $[0.5,0.25]$ and $[0.5,0.5,0.25,0.5]$. These probabilities are a separate arithmetic probe, not outputs from the trained decoder.

The lists have $2$ and $4$ targets and total NLLs $3\ln2$ and $5\ln2$ nats. Together they contain $6$ targets and $8\ln2$ nats. The correct mean is $\frac{4}{3}\ln2\approx0.924196$ nats per target; its perplexity is $2^{4/3}\approx2.519842$. Averaging the two batch means instead gives $\frac{11}{8}\ln2\approx0.953077$, which overweights each target in the short batch. Averaging the two perplexities gives $2.603421$ and is also the wrong aggregation for these six targets.

For one fixed model and one evaluated split, the general rule is

Batch index $b$ selects a batch, and $I_b$ contains its valid target occurrences. Index $i$ names one occurrence, $y_i$ is its observed next token, and $x_i$ is the permitted document-local context. The fixed learned parameters are $\theta$; $p_\theta(y_i\mid x_i)$ is the probability assigned by the unmodified evaluation distribution, without sampling filters or temperature changes.

The natural logarithm $\ln$ makes $S_b$ and total $S$ losses in nats. Counts $N_b$ and total $N$ count selected targets, including selected EOS targets and excluding padding. Mean $L$ is in nats per target; $\mathrm{PPL}$ is dimensionless. Sum the numerators and denominators before dividing, then exponentiate once. For a comparison to be interpretable, keep vocabulary, selected targets, document-context policy and numerical execution explicit and comparable.

Read the measurements at their stated scale

The model has residual width $8$, two attention heads, feed-forward width $16$, one decoder block and capacity for $8$ positions. Each seed performs $80$ scalar CPU f64 logical updates at constant rate $0.02$, combining two one-document microbatches per update. AdamW uses $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$ and zero weight decay. Global gradient clipping uses limit $1$; dropout is disabled in this training fixture. Data, tokenizer, update policy and evaluated checkpoints are fixed across seeds.

All five checkpoints, $0,10,20,40,80$, appear for every seed in the figure below. Update $20$ has the smallest validation NLL among those five observations in each run. That does not make it the optimum among all possible update counts: intermediate updates were not evaluated. No early-stopping or best-seed selection algorithm is introduced. The saved example deliberately uses the first predeclared seed, $39$, at final update $80$.

The final validation NLLs for seeds $39$, $40$ and $41$ are $4.172745$, $4.328176$ and $2.545169$ nats per target. Their arithmetic mean is $3.682030$, sample standard deviation is $0.987613$, and observed range is $2.545169$–$4.328176$ nats per target. For $K=3$ seeds, with $L_k$ denoting seed $k$‘s final validation NLL and $\bar L$ their arithmetic mean, the sample standard deviation is $s=\sqrt{\sum_{k=1}^{K}(L_k-\bar L)^2/(K-1)}$. Here $k$ indexes runs rather than token occurrences, and $s$ has the same units as $L_k$.

These numbers describe initialization variation conditional on this one split and training policy. They are not a population confidence interval or a measure of uncertainty from sampling different documents. Three fixed seeds also do not establish that a model ranking would persist on other data or seeds.

The arithmetic uses represented f64 values. The report rounds metric values to six decimal places after calculation; comparison and seed statistics use unrounded values. A finite NLL need not have a finite represented exponential for every possible input, although the reported perplexities here are finite.

Compare likelihood without making it a complete quality judgment

Short-context count models provide a useful comparison, but fitting the observed training transitions does not establish performance on separate documents. Neural language models learn distributed representations and can be compared with n-gram models through held-out likelihood and perplexity.

In A Neural Probabilistic Language Model (2003), Bengio, Ducharme, Vincent and Jauvin compare neural and n-gram language models with perplexity derived from average negative log-likelihood, separating training, validation choices and final testing. Distributed representations address limitations of short count-based contexts. This chapter retains the comparison principle, using the existing bigram as a deliberately simple reference.

Fit the existing add-one-smoothed bigram only to the two training documents and their $9$ adjacent transitions. For previous token $u$ and next token $v$, it uses $q(v\mid u)=(C_{uv}+1)/(C_u+V)$, where $C_{uv}$ counts that training transition, $C_u$ counts all training transitions leaving $u$, and $V$ is the same tokenizer vocabulary size used by the decoder. The added count prevents an unseen transition from receiving zero probability. The bigram uses only the previous token; the neural decoder can use the permitted longer prefix.

Both models score the same $9$ selected target occurrences on each split. The frozen bigram has training NLL $4.402665$ and perplexity $81.668227$; validation NLL is $5.044970$ and perplexity $155.239620$. The neural seed-$39$ model at update $80$ has lower validation NLL, $4.172745$, and perplexity $64.893323$. It beats this weak baseline while scoring worse than its own update-$20$ validation result, NLL $1.267297$ and perplexity $3.551242$. Beating a baseline does not erase deterioration during continued training.

In Language Model Evaluation Beyond Perplexity (2021), Meister and Cotterell explain that held-out perplexity is a monotonic transformation of average token log-likelihood and does not identify in detail which linguistic properties a model learned. The decoder’s evaluation must preserve target accounting and split separation, while aggregate likelihood remains narrower than a detailed account of learned linguistic properties. These constructed strings establish neither useful generated language nor broad language capabilities.

Fit the existing add-one-smoothed bigram only to training documents, then compare its probabilities with the neural decoder on the same selected targets. The bounded Rust comparison measures both models on four constructed strings. Its tokenizer, baseline, seeds and update budget are course choices, not a reproduction of either paper’s results or a competitive n-gram benchmark.

Extend one evaluator and keep its state unchanged

MetricSummary::merge adds loss sums and target counts. It then recomputes the mean and perplexity from those totals, rather than averaging the summaries’ means or perplexities.

    /// Add numerators and denominators before recomputing mean NLL and perplexity.
    pub fn merge(self, other: Self) -> Result<Self, MetricError> {
        let count = self
            .target_count
            .checked_add(other.target_count)
            .ok_or(MetricError::TargetCountOverflow)?;
        Self::from_total_nll(self.total_surprise + other.total_surprise, count)
    }

The existing fixed-batch evaluator and the padded/packed entry point share one reducer. The existing selected-target loss produces a batch mean; multiplying that mean by its valid-target count recovers the numerator for aggregation, subject to represented floating-point rounding.

/// One graph-free reduction for fixed, padded and packed prediction storage.
pub(crate) fn evaluate_loss_batches_no_grad<I, F>(
    model: &DecoderModel,
    partition: Partition,
    batches: I,
    mut compute_loss: F,
) -> Result<Evaluation, TrainerError>
where
    I: IntoIterator,
    F: FnMut(AutogradContext, I::Item) -> Result<(TensorValue, usize), TrainerError>,
{
    let before = gradient_bits(model)?;
    let mut metrics: Option<MetricSummary> = None;
    let mut batch_count = 0;
    let context = AutogradContext::no_grad();
    for (batch_index, batch) in batches.into_iter().enumerate() {
        let (loss, count) = compute_loss(context, batch)?;
        if loss.tracks_gradient() {
            return Err(TrainerError::ValidationRecordedGraph {
                partition,
                batch: batch_index,
            });
        }
        let scalar = scalar_loss(&loss)?;
        let next = MetricSummary::from_total_nll(scalar * count as f64, count)?;
        metrics = Some(match metrics {
            None => next,
            Some(total) => total.merge(next)?,
        });
        batch_count += 1;
    }
    let metrics = metrics.ok_or(TrainerError::EmptyEpoch {
        role: if partition == Partition::Validation {
            TrainerEpochRole::ValidationSelection
        } else {
            TrainerEpochRole::TrainEvaluation
        },
    })?;
    if !metrics.mean_nll().is_finite() {
        return Err(TrainerError::NonFiniteLoss {
            value: metrics.mean_nll(),
        });
    }
    if gradient_bits(model)? != before {
        return Err(TrainerError::ValidationChangedGradient);
    }
    Ok(Evaluation {
        metrics,
        batch_count,
        recorded_graphs: 0,
    })
}

no_grad controls graph recording; it does not by itself disable training dropout. This evaluator also chooses the ordinary non-dropout padded forward path. An explicit DropoutMode::Eval comparison with a nonzero configured dropout probability matches that path and leaves the dropout counter unchanged. Tests start with actual nonzero gradients and check that evaluation preserves their bits, learned values and value revisions. Evaluation performs no backward or optimizer update.

/// Evaluate padded rows, or PackedBatch::storage(), through the same model and reducer.
/// The ordinary padded forward path is evaluation-mode: it applies no dropout.
pub fn evaluate<'a>(
    model: &DecoderModel,
    partition: Partition,
    expected: &BatchBinding,
    batches: impl IntoIterator<Item = &'a VariableBatch>,
) -> Result<Evaluation, TrainerError> {
    evaluate_loss_batches_no_grad(model, partition, batches, |context, batch| {
        check_batch(batch, partition, expected)?;
        let loss = model.loss_padded_with_context(context, batch, expected)?;
        Ok((loss.mean().clone(), loss.count()))
    })
}

On the initial seed-$39$ validation fixture, separate examples, padded storage and packed storage score the same $9$ targets in $2$, $1$ and $1$ batches, respectively. All displayed means are $5.615827$ nats per target, with zero recorded graphs. The largest measured unrounded NLL difference is $1.7763568394002505\times10^{-15}$, below tolerance $10^{-12}$. The packed fixture still has two rows: $5+4$ targets cannot fit in one row of capacity $8$. One batch does not mean one row. This is a numerical comparison for the stated fixture, not universal bitwise equality across layouts or backends.

Empty evaluation, a nonfinite neural loss, and a mismatched declared partition or binding are refused. Validation examples are refused by the training-batch constructor. The lower-level probability metric retains its existing meaning for an impossible observed target: probability zero gives infinite NLL and perplexity, without a clamp. Invalid probability values are rejected. That metric convention is distinct from the neural evaluator’s nonfinite-loss refusal.

The historical contrast also reuses existing code. BigramModel is fitted once on training documents; the scorer below reads only input/target pairs whose loss-mask entry is valid. It therefore excludes padding and never joins the end of one document to the beginning of another.

/// Score the same mask-selected transitions with the existing frozen bigram.
pub fn score_bigram<'a>(
    model: &BigramModel,
    partition: Partition,
    expected: &BatchBinding,
    batches: impl IntoIterator<Item = &'a VariableBatch>,
) -> Result<MetricSummary, TrainerError> {
    let batches: Vec<_> = batches.into_iter().collect();
    for batch in &batches {
        check_batch(batch, partition, expected)?;
    }
    Ok(score_bigram_transitions(
        model,
        batches.into_iter().flat_map(|batch| {
            batch
                .inputs()
                .iter()
                .zip(batch.targets())
                .zip(batch.loss_mask())
                .filter_map(|((&input, &target), &valid)| {
                    valid.then_some((input as u32, target as u32))
                })
        }),
    )?)
}

The constructed probability probe makes the unequal denominator easy to inspect without conflating those assigned probabilities with a learned model’s output.

/// Constructed probabilities, not decoder predictions, make the denominator visible.
pub fn weighting_probe() -> DemoResult<Value> {
    use llm_from_scratch_practical::metrics::score_assigned_probabilities;
    let short = score_assigned_probabilities(&[0.5, 0.25])?;
    let long = score_assigned_probabilities(&[0.5, 0.5, 0.25, 0.5])?;
    let combined = short.merge(long)?;
    Ok(
        json!({"batch_targets":[short.target_count(),long.target_count()],
        "batch_mean_nll":[rounded(short.mean_nll()),rounded(long.mean_nll())],
        "correct":metric_record(combined),
        "incorrect_unweighted_mean_nll":rounded((short.mean_nll()+long.mean_nll())/2.0),
        "incorrect_mean_perplexity":rounded((short.perplexity()+long.perplexity())/2.0)}),
    )
}

The seed summary uses the final unrounded validation NLL from every declared run and divides the squared-deviation sum by $K-1$, with $K=3$.

fn seed_summary(values: &[f64]) -> Value {
    let mean = values.iter().sum::<f64>() / values.len() as f64;
    let variance = values
        .iter()
        .map(|value| (value - mean).powi(2))
        .sum::<f64>()
        / (values.len() - 1) as f64;
    json!({"seed_count":values.len(),"mean_nll":rounded(mean),"sample_standard_deviation_nll":rounded(variance.sqrt()),
        "minimum_nll":rounded(values.iter().copied().fold(f64::INFINITY,f64::min)),
        "maximum_nll":rounded(values.iter().copied().fold(f64::NEG_INFINITY,f64::max)),
        "interpretation":"descriptive initialization variation on this fixed constructed split; no population confidence interval"})
}

Finally, evaluate_saved restores the actual portable model and tokenizer. It encodes the chapter’s documents with those restored assets and scores their selected targets without retraining. It does not refit a tokenizer or require the original tokenizer-training metadata. The text encoder, source-independent backbone and tied head remain the same composition.

/// Reloads the actual saved encoder assets and weights; performs no training.
pub fn evaluate_saved(path: &Path, rows: usize) -> DemoResult<Value> {
    let archive = PortableModel::load(path)?;
    let (model, tokenizer) = archive.restore_text()?;
    Ok(
        json!({"chapter":"16-language-model-evaluation","execution":"scalar-cpu-f64",
        "data_scope":"the chapter's constructed train and validation fixtures",
        "model_sha256":archive.model_id()?,
        "scores":score_model(&model,&fixture_from_tokenizer(tokenizer)?,rows)?}),
    )
}

Follow the same frozen weights through both scores

The tables read the Rust fixture’s reported observations. Each column keeps the same target denominator throughout its run. Compare changes between checkpoints within a run before using the spread across runs to describe initialization variation.

Reproduce the scores and inspect the saved model

From the repository root, with Docker and the existing course Docker setup available, run the default CPU example. No GPU, external corpus or downloaded model is required; the four documents and fixed experiment settings are included.

COURSE_CORPUS=false ./course run cargo run --quiet --offline --locked -p practical-ch16-language-model-evaluation

The JSON contains weighting, layout_comparison, baseline, all three runs, final_validation_seed_summary and saved_first_seed_model. Each run contains the five checkpoints and its two observed overfitting intervals. The default saved-model comparison is an in-memory round trip.

For a real process boundary, keep the same working directory and Docker setup and run this entire block as one command. Each Cargo invocation exits before the next program starts; one container retains their saved file.

COURSE_CORPUS=false ./course run bash -euc '
mkdir -p tmp/ch16-evaluation
cargo run --quiet --offline --locked -p practical-ch16-language-model-evaluation -- experiment --save-model tmp/ch16-evaluation/model.safetensors
cargo run --quiet --offline --locked -p practical-ch16-language-model-evaluation -- evaluate --model tmp/ch16-evaluation/model.safetensors --batch-rows 1
cargo run --quiet --offline --locked -p practical-ch16-language-model-evaluation -- evaluate --model tmp/ch16-evaluation/model.safetensors --batch-rows 2
'

The producer saves seed $39$ at update $80$, not the seed with the smallest validation score. Both new evaluation programs restore model identity f485cea6228baa825076855af99cc71bad66b462c540590a4767df9064e07e2d. Their printed training NLL is $0.157051$ and validation NLL is $4.172745$; each split has $9$ targets and zero recorded graphs. One row per batch produces two batches per split; two rows produce one. The printed metrics match at six decimal places. That printed equality alone does not establish the unrounded $10^{-12}$ tolerance checked by the Rust layout tests. Separate ./course run invocations would use disposable containers and would not retain this file.

With the same repository-root working directory and Docker setup, run the demo library tests:

COURSE_CORPUS=false ./course run cargo test --offline --locked -p practical-ch16-language-model-evaluation

The library result is ten passed tests. They cover target weighting, layout agreement, preserved nonzero gradients and explicit evaluation dropout, split/binding refusals, the training-only baseline and the saved composition.

Optional inspection and explanation practice

1. Use weighting from the default command above to reproduce the six-target mean and perplexity from the two constructed probability lists. Explain why the unweighted batch mean is different.
2. Inspect every seed’s checkpoints in the default output or figure. Identify the measured intervals with falling training NLL and rising validation NLL, and explain the limit of calling update $20$ the best checkpoint.
3. Compare the frozen bigram’s validation score with seed $39$ at updates $20$ and $80$. Then inspect all three final validation NLLs and their summary. Explain what those two comparisons establish.
4. Use layout_comparison from the default command and run the ten-test command above. Explain the target counts, batch counts, two-row packed storage and distinction between no-grad recording and evaluation dropout.
5. Run the complete single-container save/evaluate command above. Compare model identity, target counts, printed scores and batch counts in the two new evaluation records.

Keep an evaluation reference for weight quantization

The same text encoder, source-independent backbone and tied head now have shared valid-target evaluation across fixed, padded and packed batches. The saved model can be scored independently on CPU. Training-only fitting, held-out scoring and the complete seed record keep each comparison tied to its actual evidence.

Planned Chapter 17, Weight quantization, will use evaluation as a reference when inspecting the numerical and held-out effects of weight rounding. It remains a future lesson. Physical-GPU and full-corpus results remain separate evidence; the four constructed strings here support only their stated bounded comparisons.