34 · Content revision 7

Open one local test gate, keep the report

Learn how one local final-evaluation gate isolates an already selected state, compare graph-free decoder and bigram scores fairly, and distinguish a deliberately selected fixed regression fixture from independent generalization evidence.

Freeze the comparison before opening test

Evaluate the frozen validation-selected decoder through one local test-only gate, validate and record the ordered test input/target positions, aggregate every target token fairly, and compare the result with a frozen bigram fitted on the same training tokens. Chapter 33 already chose checkpoint $s^*=8$ with validation loss $1.595297$ and proved that an attempted test-partition probe was rejected during selection.

Before creating the test gate, the fixture freezes every remaining choice:

vocabulary size $V=5$ and context length $T=2$ ;
the Chapter 33 selected parameter snapshot;
an add-one bigram fitted on the exact same two training token slices;
sequential stride-one windows and batch width $4$ ; and
two test documents of unequal length.

test-a = [4,3,2,1,0,4,3,2,1]
test-b = [3,2,1,0,4,3,2]

The first document contributes $14$ target-token positions and the second contributes $10$ . Each target has one aligned input position, but the loss has one term per target; aligned inputs do not double the denominator. Therefore $N_{te}=14+10=24$ , not $48$ scalar IDs. FinalEvaluator::evaluate_once opens its local gate, checks the ordered pairs, evaluates both models, verifies that every decoder parameter and gradient bit is unchanged, and only then returns the report containing

\mathcal{L}_{te}^{\mathrm{decoder}}=1.607679,\qquad \mathcal{L}_{te}^{\mathrm{bigram}}=2.236735.

Thus the decoder is lower by $\Delta_{te}=0.629055$ in this fixture. The test documents deliberately reverse the synthetic training cycle and were selected for that recorded ordering after a neutral holdout did not preserve it. The exact ordering is now rerun as a regression condition. These are valid measurements of this fixed teaching fixture and its executable boundary, but they are not an untouched independent estimate of generalization and do not establish architecture-wide decoder superiority.

Average surprise over target tokens

The complete final decoder score is

\mathcal{L}_{te}(\theta_{s^*})=-\frac{1}{N_{te}}\sum_{n=1}^{N_{te}}\log p_{\theta_{s^*}}(y_n\mid x_n)

The state $\theta_{s^*}$ is already frozen before the local gate opens. Inside this execution, test loss may describe it but cannot change $s^*$ , the learning-rate schedule, the tokenizer, the baseline, or any other choice. That within-run guarantee does not make a deliberately selected and repeatedly checked repository fixture an untouched generalization estimate.

Unequal documents require token weighting. If document $d$ contributes $N_d$ targets with mean loss $\mathcal{L}^{(d)}_{te}$ , then

\mathcal{L}_{te}=\frac{\sum_d N_d\mathcal{L}^{(d)}_{te}}{\sum_d N_d},\qquad N_{te}=\sum_d N_d

For this run, $N_1=14$ , $N_2=10$ , and $N_{te}=24$ . Averaging two document means would incorrectly give the shorter document the same weight as the longer one.

Keep states, slots, and roles distinct

The $N_{te}$ aligned target slots are not just a count. Each slot is one causal input $x_n$ paired with its observed next token $y_n$ . Its document, window, and within-window position fix its identity, and overlapping windows can repeat one underlying document transition.

$\mathcal{L}_{te}$ is token-weighted mean negative log-likelihood on Test.
$\theta_{s^*}$ is the complete decoder state chosen before test opens.
$s^*$ is the validation-selected checkpoint index, here $s^*=8$ .
$N_{te}$ is the complete test target count, here $N_{te}=24$ .
$n$ indexes one stable document, window, and within-window input/target position.
$x_n$ is the input context available for the target at position $n$ .
$y_n$ is the observed next-token target at slot $n$ .
$p_{\theta_{s^*}}(y_n\mid x_n)$ is the frozen decoder probability assigned to that target.
$\log$ is the natural logarithm, so loss is in nats per target token.

Train, Validation, and Test are concrete partition variants in the implementation. Conceptually, training fits, validation selects, and a genuinely untouched test set can supply independent final evidence. Once a test result changes a choice, that result has become selection evidence. A known result that is rerun permanently is useful regression evidence, but not a new independent measurement each time.

The local gate is deliberately not a claim of global uniqueness. Another process could construct another owner from copied data. Real evaluations also require access control, audit logs, and dataset governance.

From training scores to governed final LLM evidence

Early neural language-model evaluation moved from training-set reporting and repeatedly consulted holdouts toward a three-role protocol: training fits, validation selects, and test supplies one final comparison after every choice is frozen.

A Neural Probabilistic Language Model shows an early neural-language-model separation. Bengio and colleagues use separate training, validation, and test portions, use validation for model choices and early stopping, and then report test perplexity for an early neural language model. Training scores judge data the model already fitted, while adaptive repeated reuse of one holdout can overfit that holdout and destroys its status as untouched independent evidence. The paper supports the three roles; it does not establish an exactly-once gate-opening count.

Generalization in Adaptive Data Analysis and Holdout Reuse supplies the adaptive-use warning. Dwork and colleagues show that adaptive repeated reuse of a standard holdout can overfit the holdout itself; this general warning does not establish any fact about this repository’s fixture history or scores.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer makes checkpoint responsibility operational at larger scale. Raffel and colleagues save fine-tuning checkpoints, select by validation performance, and generally keep exploratory comparisons on validation data to avoid test-set model selection before final reporting. The selected quantity is task-dependent validation performance, not universally validation loss.

Language Models are Few-Shot Learners adds the web-scale provenance pressure. Brown and colleagues examine overlap between web-scale pretraining data and evaluation benchmarks, compare clean subsets, and omit heavily contaminated language-model tasks, showing why nominal split labels alone do not settle provenance. A nominal test label therefore does not by itself prove independence. The mostly small clean-subset changes permit multiple interpretations and do not prove contamination caused a score increase.

Neural language-model studies separated fitting, validation-driven choices, and test reporting; large transfer studies made validation checkpoint choice operationally explicit, while web-scale pretraining added dataset-overlap and provenance audits to the evaluation boundary. Adaptive-data analysis adds the warning about repeated holdout use. A trustworthy final LLM comparison freezes model and data decisions before scoring, records provenance, and distinguishes an untouched estimate from a known fixed fixture retained for regression checking.

This chapter uses a deliberately strict one-gate rule to make the within-run boundary observable. It does not claim that these papers used exactly one test query, and the local counter cannot replace organizational data governance. At repository level, the reverse-cycle documents were selected for the recorded ordering and the result is checked repeatedly. The score is therefore fixed-fixture regression evidence, not an untouched independent estimate of generalization.

Encode the modern separation between a validation-selected model and one local test evaluation rust/crates/llm-from-scratch/src/evaluation.rs#once-only-final-evaluation

/// Owns one test epoch and consumes its local scoring permission on first use.
///
/// This type is deliberately not cloneable. It enforces one access through one
/// owner; external dataset governance is still required to prevent another
/// process from constructing a separate owner over copied data. Construction
/// checks the epoch's stored `Test` enum value, not its external lineage.
#[derive(Debug)]
pub struct FinalEvaluator {
    test_epoch: MiniBatchEpoch,
    provenance: EvaluationProvenance,
    access_count: u8,
}

impl FinalEvaluator {
    pub fn new(
        test_epoch: MiniBatchEpoch,
        provenance: EvaluationProvenance,
    ) -> Result<Self, EvaluationError> {
        if test_epoch.partition() != Partition::Test {
            return Err(EvaluationError::WrongTestPartition {
                actual: test_epoch.partition(),
            });
        }
        if test_epoch.window_count() == 0 {
            return Err(EvaluationError::EmptyTestEpoch);
        }
        if test_epoch.context_length() != provenance.context_length() {
            return Err(EvaluationError::EpochContextMismatch {
                expected: provenance.context_length(),
                actual: test_epoch.context_length(),
            });
        }
        Ok(Self {
            test_epoch,
            provenance,
            access_count: 0,
        })
    }

    pub const fn access_count(&self) -> u8 {
        self.access_count
    }

    pub fn evaluate_once(
        &mut self,
        decoder: SelectedDecoder<'_>,
        bigram: FrozenBigram<'_>,
    ) -> Result<FinalEvaluationReport, EvaluationError> {
        if self.access_count != 0 {
            return Err(EvaluationError::AlreadyEvaluated);
        }
        require_matching_provenance_assertions(&self.provenance, decoder.provenance())?;
        require_matching_provenance_assertions(&self.provenance, bigram.provenance())?;

        if !selected_state_matches_model(decoder.state(), decoder.model()) {
            return Err(EvaluationError::SelectedStateMismatch);
        }

        let model_config = decoder.model().config();
        if model_config.max_positions() != self.provenance.context_length() {
            return Err(EvaluationError::ModelContextMismatch {
                expected: self.provenance.context_length(),
                actual: model_config.max_positions(),
            });
        }
        let decoder_vocabulary = model_config.vocabulary_size();
        let bigram_vocabulary = bigram.model().vocabulary_size();
        if decoder_vocabulary != bigram_vocabulary {
            return Err(EvaluationError::VocabularyMismatch {
                decoder: decoder_vocabulary,
                bigram: bigram_vocabulary,
            });
        }

        // Every metadata error above leaves the test unopened. From this line on,
        // even an error burns the local gate because token evidence is inspected.
        self.access_count = 1;
        let inspected = InspectedTestEpoch::inspect(&self.test_epoch, decoder_vocabulary)?;
        let model = decoder.model();
        let parameters_before = parameter_bits(model);
        let gradients_before = gradient_bits(model)?;

        let measured = evaluate_no_grad(model, inspected.epoch())?;
        if measured.recorded_graphs() != 0 {
            return Err(EvaluationError::GraphRecorded {
                count: measured.recorded_graphs(),
            });
        }
        let parameters_after = parameter_bits(model);
        if parameters_after != parameters_before {
            return Err(EvaluationError::DecoderParameterChanged);
        }
        let gradients_after = gradient_bits(model)?;
        if gradients_after != gradients_before {
            return Err(EvaluationError::DecoderGradientChanged);
        }

        let decoder_score = ModelScore::new(
            EvaluatedModel::SelectedDecoder,
            measured.mean_loss() * measured.token_count() as f64,
            measured.token_count(),
            measured.mean_loss(),
            measured.mean_loss().exp(),
        )?;
        let bigram_score = score_bigram(bigram.model(), &inspected)?;
        if inspected.evidence().window_target_slot_count != measured.token_count()
            || inspected.evidence().window_target_slot_count != bigram_score.target_count()
        {
            return Err(EvaluationError::TargetCountMismatch {
                expected: inspected.evidence().window_target_slot_count,
                decoder: measured.token_count(),
                bigram: bigram_score.target_count(),
            });
        }
        let evidence = inspected.into_evidence();

        Ok(FinalEvaluationReport {
            version: FINAL_EVALUATION_REPORT_VERSION,
            selected_step: decoder.selected_step(),
            selected_validation_loss: decoder.selected_validation_loss(),
            provenance: self.provenance.clone(),
            test_document_ids: evidence.document_ids,
            window_slot_fingerprint: evidence.window_slot_fingerprint,
            window_count: self.test_epoch.window_count(),
            batch_count: self.test_epoch.batch_count(),
            window_target_slot_count: evidence.window_target_slot_count,
            document_transition_occurrence_count: evidence.document_transition_occurrence_count,
            transition_multiplicity_counts: evidence.transition_multiplicity_counts,
            decoder: decoder_score,
            bigram: bigram_score,
            access_count: self.access_count,
            recorded_graphs: measured.recorded_graphs(),
            parameters_unchanged: true,
            gradients_unchanged: true,
        })
    }
}

Make the final boundary executable

EvaluationProvenance stores three nonblank strings supplied by the caller and named corpus, split, and tokenizer fingerprints, plus a positive context length. The evaluator compares those strings for exact equality; it neither derives them nor checks their relationship to the underlying corpus, split construction, or tokenizer. Equal strings can therefore describe different underlying artifacts. Context has a stronger runtime check: it must also equal the test epoch’s context length and the decoder’s maximum context length.

SelectedDecoder::new requires the caller’s selection-partition assertion to be Validation. It does not validate selected_step, and it checks selected_validation_loss only for finiteness and nonnegativity. Evaluation later verifies that the retained state and borrowed model match exactly, but it cannot reconstruct how the step or state was selected. FrozenBigram::new requires a caller-supplied Train assertion and a model that reports at least one fitted document, but it cannot discover which documents produced the counts. FinalEvaluator checks the epoch’s stored Test role; that label alone does not prove external holdout lineage.

Distinguish caller-supplied identifiers and role assertions from the context and exact state/model facts checked during evaluation rust/crates/llm-from-scratch/src/evaluation.rs#evaluation-provenance

/// Caller-supplied identifiers and context metadata shared by report participants.
///
/// Construction proves only that the three identifier strings are nonblank and
/// the context length is positive. Equality between values of this type proves
/// only that callers supplied matching assertions; it does not hash or inspect a
/// corpus, split construction, or tokenizer. The evaluator separately checks the
/// context value against the test epoch and decoder capacity.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct EvaluationProvenance {
    corpus_fingerprint: String,
    split_fingerprint: String,
    tokenizer_fingerprint: String,
    context_length: usize,
}

impl EvaluationProvenance {
    pub fn new(
        corpus_fingerprint: impl Into<String>,
        split_fingerprint: impl Into<String>,
        tokenizer_fingerprint: impl Into<String>,
        context_length: usize,
    ) -> Result<Self, EvaluationError> {
        let provenance = Self {
            corpus_fingerprint: corpus_fingerprint.into(),
            split_fingerprint: split_fingerprint.into(),
            tokenizer_fingerprint: tokenizer_fingerprint.into(),
            context_length,
        };
        for (field, value) in [
            (ProvenanceField::Corpus, provenance.corpus_fingerprint()),
            (ProvenanceField::Split, provenance.split_fingerprint()),
            (
                ProvenanceField::Tokenizer,
                provenance.tokenizer_fingerprint(),
            ),
        ] {
            if value.trim().is_empty() {
                return Err(EvaluationError::EmptyProvenance { field });
            }
        }
        if context_length == 0 {
            return Err(EvaluationError::ZeroContextLength);
        }
        Ok(provenance)
    }

    pub fn corpus_fingerprint(&self) -> &str {
        &self.corpus_fingerprint
    }

    pub fn split_fingerprint(&self) -> &str {
        &self.split_fingerprint
    }

    pub fn tokenizer_fingerprint(&self) -> &str {
        &self.tokenizer_fingerprint
    }

    pub const fn context_length(&self) -> usize {
        self.context_length
    }
}

/// A borrowed decoder accompanied by the caller's validation-role assertion.
///
/// Construction checks the asserted role and the numeric shape of the supplied
/// selection values. It cannot reconstruct how `selected_step`,
/// `selected_validation_loss`, or the model were selected. Before test access,
/// the evaluator does mechanically compare the retained state with the borrowed
/// model's exact configuration, names, shapes, and value bits.
#[derive(Clone, Copy, Debug)]
pub struct SelectedDecoder<'a> {
    state: &'a DecoderModelState,
    model: &'a DecoderModel,
    selected_step: usize,
    selected_validation_loss: f64,
    provenance: &'a EvaluationProvenance,
}

impl<'a> SelectedDecoder<'a> {
    pub fn new(
        state: &'a DecoderModelState,
        model: &'a DecoderModel,
        selected_step: usize,
        selected_validation_loss: f64,
        selection_partition_assertion: Partition,
        provenance: &'a EvaluationProvenance,
    ) -> Result<Self, EvaluationError> {
        if selection_partition_assertion != Partition::Validation {
            return Err(EvaluationError::WrongSelectionPartition {
                actual: selection_partition_assertion,
            });
        }
        if !selected_validation_loss.is_finite() || selected_validation_loss < 0.0 {
            return Err(EvaluationError::InvalidSelectionLoss {
                value: selected_validation_loss,
            });
        }
        Ok(Self {
            state,
            model,
            selected_step,
            selected_validation_loss,
            provenance,
        })
    }

    const fn model(self) -> &'a DecoderModel {
        self.model
    }

    const fn state(self) -> &'a DecoderModelState {
        self.state
    }

    pub const fn selected_step(self) -> usize {
        self.selected_step
    }

    pub const fn selected_validation_loss(self) -> f64 {
        self.selected_validation_loss
    }

    pub const fn provenance(self) -> &'a EvaluationProvenance {
        self.provenance
    }
}

/// A borrowed count baseline accompanied by the caller's training-role assertion.
///
/// Construction checks the asserted role and that the model reports at least one
/// fitted document. It cannot discover which documents produced the counts.
#[derive(Clone, Copy, Debug)]
pub struct FrozenBigram<'a> {
    model: &'a BigramModel,
    provenance: &'a EvaluationProvenance,
}

impl<'a> FrozenBigram<'a> {
    pub fn new(
        model: &'a BigramModel,
        fit_partition_assertion: Partition,
        provenance: &'a EvaluationProvenance,
    ) -> Result<Self, EvaluationError> {
        if fit_partition_assertion != Partition::Train {
            return Err(EvaluationError::WrongBaselinePartition {
                actual: fit_partition_assertion,
            });
        }
        if model.fitted_documents() == 0 {
            return Err(EvaluationError::UnfittedBigram);
        }
        Ok(Self { model, provenance })
    }

    pub const fn model(self) -> &'a BigramModel {
        self.model
    }

    pub const fn provenance(self) -> &'a EvaluationProvenance {
        self.provenance
    }
}

The Chapter 34 fixture supplies the intended histories at its assembly call sites. TrainingResult provides the actual Chapter 33 retained selected state and matching decoder. The fixture obtains Chapter 33’s exact training slices through a read-only helper and fits the unchanged alpha-one bigram before it constructs the test evaluator. That concrete assembly evidence is stronger than the generic constructors’ labels; external datasets still require trusted fingerprint derivation, access controls, audit logs, and dataset governance.

That assembly history also fixes the evidence scope. The reverse-cycle documents were deliberately selected for the recorded decoder-lower ordering, which the learner program now retains as a regression condition. This does not alter the generic evaluator: an alternate fixed-sequence diagnostic makes the bigram lower while the same graph-free and unchanged-state guarantees still pass.

Borrow the retained selected state and matching decoder, then freeze the same-training-token baseline before opening the unequal test documents through one local gate rust/demos/ch34-final-evaluation/src/lib.rs#learner-evidence

#[derive(Clone, Debug, PartialEq)]
pub struct LearnerEvidence {
    pub report: FinalEvaluationReport,
    pub selection_test_partition_rejected: bool,
    pub gate_openings_before: u8,
    pub baseline_alpha: f64,
    pub baseline_documents: usize,
    pub baseline_transitions: u64,
    pub token_weighted: bool,
    pub provenance_assertions_match: bool,
    pub within_run_selection_isolated: bool,
}

/// Builds both frozen candidates before opening one owned test evaluator.
pub fn learner_evidence() -> Result<LearnerEvidence, FixtureError> {
    let selected = selection_evidence()?;
    require(
        selected.test_partition_rejected,
        "selection no longer rejects the test partition",
    )?;
    let provenance_assertions = fixture_provenance_assertions()?;
    let training_documents = fixture_training_documents();
    let baseline = BigramModel::fit_training_documents(
        VOCABULARY_SIZE,
        BIGRAM_ALPHA,
        training_documents.iter().map(|(_, tokens)| *tokens),
    )?;
    require(
        baseline.fitted_documents() == training_documents.len(),
        "baseline training document count changed",
    )?;
    require(
        baseline.fitted_transitions() == 22,
        "baseline training transition count changed",
    )?;

    let decoder = SelectedDecoder::new(
        selected.result.selected_state(),
        selected.result.selected_model(),
        selected.result.selected_step(),
        selected.result.selected_validation_loss(),
        Partition::Validation,
        &provenance_assertions,
    )?;
    let bigram = FrozenBigram::new(&baseline, Partition::Train, &provenance_assertions)?;
    let mut evaluator = FinalEvaluator::new(test_epoch()?, provenance_assertions.clone())?;
    let gate_openings_before = evaluator.access_count();
    require(gate_openings_before == 0, "test gate opened before scoring")?;
    let report = evaluator.evaluate_once(decoder, bigram)?;

    require(report.version() == 1, "report version changed")?;
    require(
        report.access_count() == 1,
        "test gate-opening count changed",
    )?;
    require(report.target_count() == 24, "test target count changed")?;
    require(report.window_count() == 12, "test window count changed")?;
    require(report.batch_count() == 3, "test batch count changed")?;
    require(
        report.test_document_ids() == ["test-a", "test-b"],
        "test document order changed",
    )?;
    require(
        report.target_fingerprint() == "fnv1a64:dac4bb4d76beeb59",
        "test evidence fingerprint changed",
    )?;
    require(
        report.decoder().target_count() == report.bigram().target_count(),
        "models did not score identical target counts",
    )?;
    require(
        report.recorded_graphs() == 0
            && report.parameters_unchanged()
            && report.gradients_unchanged(),
        "graph-free state-preservation proof changed",
    )?;
    require(
        report.decoder_has_lower_loss(),
        "fixed Chapter 34 fixture no longer records lower decoder loss than its bigram",
    )?;
    let token_weighted = [report.decoder(), report.bigram()]
        .into_iter()
        .all(|score| {
            (score.total_nll() / score.target_count() as f64 - score.mean_nll()).abs() <= 1e-12
        });
    require(token_weighted, "model scores are no longer token weighted")?;
    let provenance_assertions_match = report.provenance() == &provenance_assertions;
    require(
        provenance_assertions_match,
        "report provenance assertions no longer match the fixture assertions",
    )?;
    let within_run_selection_isolated = selected.test_partition_rejected
        && gate_openings_before == 0
        && report.access_count() == 1
        && report.recorded_graphs() == 0
        && report.parameters_unchanged()
        && report.gradients_unchanged();
    require(
        within_run_selection_isolated,
        "within-run selection isolation evidence changed",
    )?;

    Ok(LearnerEvidence {
        report,
        selection_test_partition_rejected: selected.test_partition_rejected,
        gate_openings_before,
        baseline_alpha: baseline.alpha(),
        baseline_documents: baseline.fitted_documents(),
        baseline_transitions: baseline.fitted_transitions(),
        token_weighted,
        provenance_assertions_match,
        within_run_selection_isolated,
    })
}

FinalEvaluator owns a nonempty epoch labeled Test and is neither cloneable nor copyable. Role-assertion, provenance-assertion, selected-state/model, context, and vocabulary errors leave the gate-opening count at $0$ . After those checks pass, the evaluator changes the count to $1$ before inspecting the first test ID. A later alignment, token-range, or numerical error therefore consumes that local permission. The decoder is not consumed; the one-use resource is permission to inspect this test epoch.

InspectedTestEpoch and its fields are private to the evaluation module. The current module constructs the type only through inspect, exposes no mutation API, and keeps an immutable borrow of the original epoch. Its document IDs, ordered fingerprint, target count, and checked [input_index, target_index] pairs therefore remain coupled to that epoch. For each batch, input/target lengths must match before IDs are checked. Positions are then checked in flat order, with the input ID before the target ID at the same position. Because preflight already proved that decoder and bigram vocabulary sizes match, every stored index is valid for both models. The view stores two indices for each of the $N_{te}$ target positions, so this reusable order costs $O(N_{te})$ space.

At gate opening, validate test alignment and token bounds and record the resulting ordered pairs rust/crates/llm-from-scratch/src/evaluation.rs#inspected-test-epoch

#[derive(Debug)]
struct TestEvidence {
    document_ids: Vec<String>,
    window_slot_fingerprint: String,
    window_target_slot_count: usize,
    document_transition_occurrence_count: usize,
    transition_multiplicity_counts: Vec<usize>,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
struct CheckedTokenPair {
    input: usize,
    target: usize,
}

/// One gate-opening inspection of a borrowed test epoch.
///
/// This type is private to the module. Current callers construct it only through
/// `inspect`; it exposes no mutation API and retains an immutable epoch borrow.
#[derive(Debug)]
struct InspectedTestEpoch<'a> {
    epoch: &'a MiniBatchEpoch,
    evidence: TestEvidence,
    checked_pairs: Vec<CheckedTokenPair>,
}

fn append_checked_aligned_tokens(
    batch: usize,
    inputs: &[u32],
    targets: &[u32],
    vocabulary_size: usize,
    checked_pairs: &mut Vec<CheckedTokenPair>,
) -> Result<(), EvaluationError> {
    if inputs.len() != targets.len() {
        return Err(EvaluationError::TargetAlignmentMismatch {
            batch,
            inputs: inputs.len(),
            targets: targets.len(),
        });
    }
    for (position, (&input, &target)) in inputs.iter().zip(targets).enumerate() {
        let Some(input_index) = usize::try_from(input)
            .ok()
            .filter(|id| *id < vocabulary_size)
        else {
            return Err(EvaluationError::InputTokenOutOfRange {
                batch,
                position,
                id: input,
                vocabulary_size,
            });
        };
        let Some(target_index) = usize::try_from(target)
            .ok()
            .filter(|id| *id < vocabulary_size)
        else {
            return Err(EvaluationError::TargetTokenOutOfRange {
                batch,
                position,
                id: target,
                vocabulary_size,
            });
        };
        checked_pairs.push(CheckedTokenPair {
            input: input_index,
            target: target_index,
        });
    }
    Ok(())
}

impl<'a> InspectedTestEpoch<'a> {
    fn inspect(epoch: &'a MiniBatchEpoch, vocabulary_size: usize) -> Result<Self, EvaluationError> {
        let mut document_ids = Vec::new();
        let mut window_target_slot_count = 0_usize;
        let mut transition_multiplicities = BTreeMap::<(String, usize), usize>::new();
        let mut fingerprint = 14_695_981_039_346_656_037_u64;
        let checked_pair_count = epoch
            .batches()
            .iter()
            .map(|batch| batch.targets().len())
            .sum();
        let mut checked_pairs = Vec::with_capacity(checked_pair_count);

        for (batch_index, batch) in epoch.batches().iter().enumerate() {
            append_checked_aligned_tokens(
                batch_index,
                batch.inputs(),
                batch.targets(),
                vocabulary_size,
                &mut checked_pairs,
            )?;
            for row in 0..batch.batch_width() {
                let origin = &batch.provenance()[row];
                if !document_ids
                    .iter()
                    .any(|existing| existing == origin.document_id())
                {
                    document_ids.push(origin.document_id().to_owned());
                }
                fnv1a_bytes(&mut fingerprint, origin.document_id().as_bytes());
                fnv1a_byte(&mut fingerprint, 0xff);
                fnv1a_bytes(&mut fingerprint, &(origin.start() as u64).to_le_bytes());
                let inputs = batch
                    .input_row(row)
                    .expect("batch row is constructed from complete inputs");
                let targets = batch
                    .target_row(row)
                    .expect("batch row is constructed from complete targets");
                for (slot, (&input, &target)) in inputs.iter().zip(targets).enumerate() {
                    fnv1a_bytes(&mut fingerprint, &input.to_le_bytes());
                    fnv1a_bytes(&mut fingerprint, &target.to_le_bytes());
                    window_target_slot_count += 1;
                    let absolute_target_position = origin.start() + slot + 1;
                    *transition_multiplicities
                        .entry((origin.document_id().to_owned(), absolute_target_position))
                        .or_default() += 1;
                }
            }
        }

        let document_transition_occurrence_count = transition_multiplicities.len();
        let maximum_multiplicity = transition_multiplicities
            .values()
            .copied()
            .max()
            .unwrap_or(0);
        let mut transition_multiplicity_counts = vec![0_usize; maximum_multiplicity];
        for multiplicity in transition_multiplicities.into_values() {
            transition_multiplicity_counts[multiplicity - 1] += 1;
        }
        debug_assert_eq!(window_target_slot_count, checked_pairs.len());
        debug_assert_eq!(
            window_target_slot_count,
            transition_multiplicity_counts
                .iter()
                .enumerate()
                .map(|(index, count)| (index + 1) * count)
                .sum::<usize>()
        );
        Ok(Self {
            epoch,
            evidence: TestEvidence {
                document_ids,
                window_slot_fingerprint: format!("fnv1a64:{fingerprint:016x}"),
                window_target_slot_count,
                document_transition_occurrence_count,
                transition_multiplicity_counts,
            },
            checked_pairs,
        })
    }

    const fn epoch(&self) -> &'a MiniBatchEpoch {
        self.epoch
    }

    const fn evidence(&self) -> &TestEvidence {
        &self.evidence
    }

    fn checked_pairs(&self) -> &[CheckedTokenPair] {
        &self.checked_pairs
    }

    fn into_evidence(self) -> TestEvidence {
        self.evidence
    }
}

evaluate_no_grad still traverses the original epoch separately for decoder scoring. That call records no graph, and the evaluator compares parameter and gradient bits before and after it. Bigram scoring accepts only the private inspected view and uses its stored index pairs, including repetitions from overlapping decoder windows. Bigram scoring never revalidates the original token arrays.

The crate-private bigram primitive trusts those stored indices and uses the same numerator and denominator arithmetic as the public probability method. The public BigramModel::smoothed_probability entry accepts two raw u32 IDs and checks both; other public raw-ID entries retain their existing checks.

This establishes one input-validation boundary for report evidence and later bigram scoring. It does not promise one physical memory pass or remove checks needed by decoder evaluation. Evidence collection, separate no-grad decoder scoring, and bigram scoring still traverse the information needed for their distinct jobs. Here “input validation” means checking lengths and token bounds; the Validation partition is the different, earlier stage that selected checkpoint $s^*$ .

The report has getters but no setters or selection operation. It owns schema version $1$ , the selected step, caller-supplied provenance assertions, ordered target fingerprint, counts, both model scores, one gate-opening count, and state-preservation checks. The learner output then states the interpretation explicitly:

evidence=scope:fixed-fixture-regression within_run_selection_isolated:true fixture_selected_for_ordering:true independent_generalization_estimate:false architecture_superiority_evidence:false

The diagram trace calls the numeric relation decoder_lower_on_fixture=true; the stale name decoder_beats_bigram is not current evidence.

Keep public raw-token calls checked while final evaluation reuses already-checked indices rust/crates/llm-from-scratch/src/bigram.rs#checked-bigram-probability

    /// Scores indices whose vocabulary bounds were established by a crate-owned boundary.
    pub(crate) fn smoothed_probability_for_checked_indices(
        &self,
        from: usize,
        to: usize,
    ) -> Result<f64, BigramError> {
        debug_assert!(from < self.vocabulary_size);
        debug_assert!(to < self.vocabulary_size);
        let row_start = from * self.vocabulary_size;
        let numerator = self.counts[row_start + to] as f64 + self.alpha;
        let row_total = self.counts[row_start..row_start + self.vocabulary_size]
            .iter()
            .try_fold(0_u64, |total, count| {
                total
                    .checked_add(*count)
                    .ok_or(BigramError::TooManyTransitions)
            })?;
        let denominator = row_total as f64 + self.alpha * self.vocabulary_size as f64;
        Ok(numerator / denominator)
    }

Run the fixed fixture through its local gate and print only its formatted immutable-report evidence rust/demos/ch34-final-evaluation/src/main.rs

fn main() {
    print!(
        "{}",
        ch34_final_evaluation::learner_report().expect("Chapter 34 fixture must remain valid")
    );
}

Run cargo run --quiet --locked -p ch34-final-evaluation. The printed report shows the rejected selection-time probe, the gate-opening count before and after evaluation, both token-weighted scores, the state-preservation checks, and the fixed-fixture evidence scope.

Read one information boundary and one comparison

The evidence record keeps the gate result, provenance assertions, inspected target order, both scores, shared target count, mechanically checked state-preservation facts, and the fixed-fixture regression scope together.

Record the exact gate, provenance assertions, scores, comparison, and mechanically checked values rust/demos/ch34-final-evaluation/src/diagram_trace.rs#final-evaluation-trace

/// Emits the exact static evidence consumed by the Chapter 34 figure.
pub fn diagram_trace() -> Result<String, FixtureError> {
    let evidence = learner_evidence()?;
    let report = &evidence.report;
    Ok(format!(
        "FINAL_EVALUATION_TRACE_V1\n\
REPORT|version={}|partition=test|selected_step={}|selection_criterion=validation-only|gate_openings_before={}|gate_openings_after={}\n\
GATE|selection_test_partition_rejected={}\n\
PROVENANCE|corpus={}|split={}|tokenizer={}|vocabulary={}|context={}|documents={}|windows={}|batches={}|targets={}|target_fingerprint={}\n\
SCORE|model=selected-decoder|fit_partition=train|selected_by=validation|targets={}|total_nll={:.6}|mean_nll={:.6}|perplexity={:.6}\n\
SCORE|model=frozen-bigram|fit_partition=train|selected_by=none|targets={}|total_nll={:.6}|mean_nll={:.6}|perplexity={:.6}\n\
COMPARE|lower_loss=selected-decoder|loss_gap={:.6}|same_targets=true|decoder_lower_on_fixture={}|evidence_scope={}|within_run_selection_isolated={}|fixture_selected_for_ordering={}|independent_generalization_estimate={}|architecture_superiority_evidence={}\n\
PROOF|token_weighted={}|provenance_assertions_match={}|graph_nodes={}|parameters_unchanged={}|gradients_unchanged={}|selection_closed={}\n\
END_FINAL_EVALUATION_TRACE\n",
        report.version(),
        report.selected_step(),
        evidence.gate_openings_before,
        report.access_count(),
        evidence.selection_test_partition_rejected,
        report.provenance().corpus_fingerprint(),
        report.provenance().split_fingerprint(),
        report.provenance().tokenizer_fingerprint(),
        VOCABULARY_SIZE,
        report.provenance().context_length(),
        report.test_document_ids().join(","),
        report.window_count(),
        report.batch_count(),
        report.target_count(),
        report.target_fingerprint(),
        report.decoder().target_count(),
        report.decoder().total_nll(),
        report.decoder().mean_nll(),
        report.decoder().perplexity(),
        report.bigram().target_count(),
        report.bigram().total_nll(),
        report.bigram().mean_nll(),
        report.bigram().perplexity(),
        report.loss_gap(),
        report.decoder_has_lower_loss(),
        FIXED_FIXTURE_EVIDENCE_SCOPE,
        evidence.within_run_selection_isolated,
        FIXTURE_SELECTED_FOR_ORDERING,
        INDEPENDENT_GENERALIZATION_ESTIMATE,
        ARCHITECTURE_SUPERIORITY_EVIDENCE,
        evidence.token_weighted,
        evidence.provenance_assertions_match,
        report.recorded_graphs(),
        report.parameters_unchanged(),
        report.gradients_unchanged(),
        evidence.selection_test_partition_rejected,
    ))
}

Separate local isolation from fixture evidence

Follow training and validation to one local test gate. The evaluator records 24 ordered input/target pairs, while explicit scope cues identify the deliberately selected decoder-lower-than-bigram loss ordering as fixed-fixture regression evidence rather than independent generalization or architecture superiority.

Equivalence sign: same inspected target order
Double border: lower loss on the fixed fixture
Cross: selection rejected the test partition

Give each partition one responsibility

The numbered sequence assigns one responsibility to each stage: training fits, validation selects, and the evaluator may inspect test IDs only after every choice is frozen.

Train fits parameters

May update parameters
Validation selects the checkpoint

May choose among planned states
Freeze every decision

No choice may change now
Open one local test gate

Validate and store ordered pairs for evidence, never selection
Keep the report immutable

Mechanically checked or recorded

Score the same inspected target order

The decoder evaluates the epoch separately without a graph, while the bigram reuses the same 24 checked input/target pairs. The double border marks only the lower recorded mean on this fixed regression fixture.

Scrollable fixed-fixture decoder and bigram scores over the same inspected target order
Model	Asserted fit role	Asserted selection role	Shared targets	Total NLL	Mean test loss
Selected decoder Double border: lower loss on the fixed fixture	Training	Validation	$N_{te}=24$	$\sum_n(-\log p_n)=38.584306$	$\mathcal{L}_{te}=1.607679$
Frozen bigram	Training	Not selected	$N_{te}=24$	$\sum_n(-\log p_n)=53.681634$	$\mathcal{L}_{te}=2.236735$

Separate assertions from checked facts

Corpus, split, and tokenizer strings are caller-supplied; equality checks only their consistency. Context, vocabulary, test targets, state/model identity, and no-grad state preservation are independently checked. The fixture assembly supplies the intended histories.

Caller-supplied identifiers agree

corpus=ch33-34-synthetic-v1 split=fixed-role-split-v1 tokenizer=literal-u32-v1 provenance_assertions_match=true

Selection was already closed

$s^*=8$

selection_test_partition

selection_test_partition_rejected=true

Decoder scoring stays separate and graph-free

$\#\mathrm{graphs}=0$ Mechanically checked or recorded

graph_nodes=0 parameters_unchanged=true gradients_unchanged=true

One inspected test view and one report

$V=5,\;T=2,\;N_{te}=24$

fnv1a64:dac4bb4d76beeb59 gate_openings_before=0 gate_openings_after=1 report_version=1

The five numbered cards show why test cannot appear before the frozen boundary inside this execution. The table repeats $N_{te}=24$ for both rows and marks the lower fixed-fixture loss with text plus a double start border. The four check cards retain the rejected selection-time test probe, one gate opening, zero recorded graph nodes, unchanged state bits, one inspected view, provenance_assertions_match=true, and fnv1a64:dac4bb4d76beeb59 target evidence.

Classify legal decisions before you run

Two documents contribute $14$ and $10$ targets. Predict the correct denominator and explain why the mean of two document means is wrong.
A developer lowers the learning rate after seeing test loss. Is that legal final reporting or a leak?
Another checkpoint has lower test loss than $\theta_{s^*}$ . May the report replace $s^*$ ?
The decoder scores overlapping context-two windows, but the bigram scores each original transition once. Are the resulting means comparable?
First change the tokenizer mapping while reusing the same tokenizer fingerprint string and the same vocabulary/context sizes. Then change only the fingerprint string. Which change can this API detect before opening?
Put token ID $5$ into this vocabulary- $5$ test epoch. What happens when the gate-opening inspection reaches it?
From losses $1.607679$ and $2.236735$ , state the justified fixed-fixture conclusion and explain why neither independent generalization nor universal architecture superiority follows.
Two processes each build a fresh local evaluator over copied test data. What external control is missing?

Misconception: gate_openings_after=1 means the repository has used this result only once. The count belongs to one evaluator instance inside one execution. It does not erase the deliberate fixture-selection history or the permanent regression reuse of the known ordering.

Check your reasoning

$N_{te}=14+10=24$ ; each target, not each document, receives equal weight.
It leaks test information into optimization; validation should have driven that choice.
No. $s^*$ was already chosen by validation. A new untouched test set would be needed after reselection.
No. Both models must score the same ordered slots, including repetitions.
Reusing the same string hides the first change from these assertion checks; the API does not inspect the tokenizer. Changing only the string creates an assertion mismatch, so the gate stays closed with count $0$ .
The inspector returns a token-range error. This FinalEvaluator has already consumed its one-use permission, so a retry returns AlreadyEvaluated.
The selected decoder is lower on this deliberately selected reversed synthetic fixture. That ordering is useful as a regression condition, but it is not an untouched independent estimate of generalization and does not show that decoder architectures are universally superior to bigrams.
Dataset access control and a shared audit log are missing.

Carry the trainer-selected model and optimizer capture forward

The cumulative decoder now has one validation-selected model state and one immutable fixed-fixture regression report on shared targets. The local evaluation cannot change that state, but the report is not an independent generalization estimate. Chapter 35 will serialize the trainer-issued selected training state: the selected model snapshot, its matching AdamW snapshot, and their shared step. It will also store the tokenizer and decoder configuration needed to interpret the model, plus a separate RNG state for later sampling. The Chapter 34 evaluation report and test provenance are not serialized, and the sampling RNG is not evaluation provenance.

Serialization must reproduce this model and its logits, not reopen test data or turn the final report into a new selection signal.