Keep a fixed reference before extending the decoder

The problem: changes need a fixed comparison point

The final chapter of LLM from scratch joined the tokenizer, decoder, training, evaluation, checkpoint and generation code. This practical course starts from a separate copy of that implementation. If the data, configuration and code change together, a changed result no longer tells you which version you observed. You need a fixed reference that identifies its inputs and states exactly what its checks establish.

This chapter runs the completed first-course pipeline without modifying it. The practical copy can then evolve while this comparison point stays fixed. The example gives that reference an input identity and a report whose claims stay attached to their evidence. The identity identifies the fixed inputs. The checks establish which behavior this execution reproduced. Neither establishes useful laptop-scale capability.

What the reference actually does

The fixture assigns $8$ documents to training, $2$ to validation and $2$ to test. The decoder has $1188$ learned scalar parameters and context capacity $4$ tokens. Its primary training schedule and separately executed replay each perform $32$ updates on full batches of $16$ windows, with $4$ valid targets per window. Therefore, each schedule consumes

valid training targets. One pipeline invocation includes both schedules: $4096$ targets altogether. These counts are derived from the frozen full-batch schedule, not from a new runtime token counter. Evaluation targets and generated tokens are separate quantities.

The report’s independent_schedules field counts two separately executed copies of the same schedule from training seed $39$, not statistically independent seed samples.

The replay agrees bit for bit with the primary training result in this execution. The selected decoder and frozen bigram then score the same $1744$ overlapping test-window target slots. Their mean negative log-likelihoods are approximately $3.866087547$ and $3.981342714$ nats per slot, respectively. This known comparison is a regression check: rerunning it can expose changes in the checked behavior. It is not an independent estimate of generalization.

There are only $442$ within-document transition occurrences in those test documents. Overlapping windows repeat occurrences, so $1744$ is not a count of distinct transitions. A separate policy would score each occurrence once, using the longest available causal prefix capped at four tokens and only its newest-position distribution. This run does not report numeric mean NLL or perplexity under that policy.

The checkpoint round trip preserves its encoded bytes, model and optimizer bits, and tokenizer. The restored decoder reproduces logits for the fixed At probe, encoded as token IDs [67,118]. Cached and complete-prefix generation agree on token IDs [260,34,34]: Cyrillic т followed by two spaces. Agreement checks the two generation paths; those three tokens do not demonstrate useful language quality.

The solution: identify inputs, then interpret evidence

The identity’s symbols and fixed settings

$R_{\mathrm{ref}}$ is a $256$-bit digest, displayed as $64$ lowercase hexadecimal characters. Its three named inputs are:

• $\mathrm{config}_{1188}$: a canonical record of eighteen tiny-run settings, together with the invariant that the decoder has $1188$ parameters.
• $\mathrm{fixture}$: a canonical record of SHA-256 digests of the exact corpus bytes and exact split-manifest bytes. Even a byte change that leaves parsed JSON meaning unchanged creates a different input identity.
• $\mathrm{source\_revision}$: a verified source identity combining a baseline revision with the exact forty Rust source files of the completed first-course crate, including its module declarations in lib.rs.

Here $\Vert$ means composition into the shared identity format’s named, canonical fields, not writing three unframed strings beside one another. That distinction matters: without field boundaries, different input sequences can have the same concatenated bytes. The helper orders fields, serializes one unambiguous record and hashes its complete bytes.

The settings fix eight BPE merges; decoder width, feed-forward width and context capacity of four; one attention head and one block; stride one; training and evaluation batch sizes of sixteen and one hundred twenty-eight; thirty-two updates; learning rate $0.04$; gradient-norm cap $1$; training seed $39$; generation seed $38$; bigram smoothing $1$; generation temperature $0.8$; top-k four; and three generated tokens. The identity records floating-point settings by their exact bit patterns rather than rounded display text.

/// Bind every tiny-config field, including exact represented floating values.
/// The count is the reference decoder's required 1,188-parameter invariant.
pub fn tiny_config_identity() -> Result<ArtifactIdentity, IdentityError> {
    let config = CapstoneConfig::tiny();
    let fields = [
        ("parameters", "1188".to_owned()),
        ("bpe_merges", config.bpe_merges().to_string()),
        ("model_width", config.model_width().to_string()),
        ("heads", config.heads().to_string()),
        (
            "feed_forward_width",
            config.feed_forward_width().to_string(),
        ),
        ("layers", config.layers().to_string()),
        ("context_length", config.context_length().to_string()),
        ("window_stride", config.window_stride().to_string()),
        ("update_batch_size", config.update_batch_size().to_string()),
        (
            "evaluation_batch_size",
            config.evaluation_batch_size().to_string(),
        ),
        ("updates", config.updates().to_string()),
        (
            "learning_rate_f64_bits",
            format!("{:016x}", config.learning_rate().to_bits()),
        ),
        (
            "max_gradient_norm_f64_bits",
            format!("{:016x}", config.max_gradient_norm().to_bits()),
        ),
        ("seed", config.seed().to_string()),
        ("generation_seed", config.generation_seed().to_string()),
        (
            "bigram_alpha_f64_bits",
            format!("{:016x}", config.bigram_alpha().to_bits()),
        ),
        (
            "generation_temperature_f64_bits",
            format!("{:016x}", config.generation_temperature().to_bits()),
        ),
        ("generation_top_k", config.generation_top_k().to_string()),
        ("generation_tokens", config.generation_tokens().to_string()),
    ];
    ArtifactIdentity::new(
        "scalar_reference_config",
        fields.iter().map(|(key, value)| (*key, value.as_str())),
    )
}

/// Pure identity construction, also usable by mutation tests without training.
/// An identity by itself does not admit alternate inputs to the reference run.
pub fn reference_input_identity(
    config: &ArtifactIdentity,
    corpus: &[u8],
    splits: &[u8],
    source: &ArtifactIdentity,
) -> Result<ArtifactIdentity, IdentityError> {
    let corpus_digest = sha256(corpus);
    let splits_digest = sha256(splits);
    let fixture = ArtifactIdentity::new(
        "scalar_reference_fixture",
        [
            ("corpus_sha256", corpus_digest.as_hex()),
            ("splits_sha256", splits_digest.as_hex()),
        ],
    )?;
    let config_bytes = canonical_string(config)?;
    let fixture_bytes = canonical_string(&fixture)?;
    let source_bytes = canonical_string(source)?;
    ArtifactIdentity::new(
        "scalar_reference_run",
        [
            ("config_1188", config_bytes.as_str()),
            ("fixture", fixture_bytes.as_str()),
            ("source_revision", source_bytes.as_str()),
        ],
    )
}

The example verifies its compiled reference-source evidence first. It then rejects changed corpus or split bytes before training or checkpoint work. Only accepted fixed inputs reach the existing pipeline. The wrapper calls that pipeline once; the pipeline itself performs primary training and replay.

/// Refuse changed fixtures before model/checkpoint work, then call the existing
/// capstone once. That call already includes primary training and full replay.
pub fn run_reference_handoff(
    corpus_source: &str,
    split_source: &str,
    checkpoint_path: impl AsRef<Path>,
    source: &ReferenceSourceProof,
) -> Result<ReferenceHandoffTrace, ReferenceHandoffError> {
    if sha256(corpus_source.as_bytes()).as_hex() != CORPUS_SHA256 {
        return Err(ReferenceHandoffError::FixtureMismatch("corpus"));
    }
    if sha256(split_source.as_bytes()).as_hex() != SPLITS_SHA256 {
        return Err(ReferenceHandoffError::FixtureMismatch("split manifest"));
    }
    // Safe callers obtain this opaque proof only through the foundation's
    // approved-manifest/compiled-census verification, not a revision string.
    let identity = reference_input_identity(
        &tiny_config_identity()?,
        corpus_source.as_bytes(),
        split_source.as_bytes(),
        source.identity(),
    )?;
    let digest = identity.digest()?;
    let observation = run_capstone(
        corpus_source,
        split_source,
        checkpoint_path,
        CapstoneConfig::tiny(),
    )?;
    let evaluation = observation.final_evaluation();
    let checkpoint = observation.checkpoint();
    if observation.training().parameter_count() != 1188
        || observation.training().optimizer_step() != 32
        || !observation.training().replay_bitwise()
        || evaluation.window_count() != 436
        || evaluation.window_target_slot_count() != 1744
        || evaluation.document_transition_occurrence_count() != 442
        || !evaluation.decoder().mean_nll().is_finite()
        || !evaluation.bigram().mean_nll().is_finite()
        || !checkpoint.bytes_roundtrip()
        || !checkpoint.model_bits_exact()
        || !checkpoint.optimizer_bits_exact()
        || !checkpoint.tokenizer_exact()
        || !checkpoint.prompt_logits_bitwise()
        || !observation.generation().tokens_exact()
        || !observation.generation().decisions_bitwise()
        || !observation.generation().rng_state_exact()
    {
        return Err(ReferenceHandoffError::Invariant(
            "scalar reference handoff evidence changed",
        ));
    }
    Ok(ReferenceHandoffTrace {
        identity,
        digest,
        source_revision: source.source_revision().to_owned(),
        observation,
    })
}

What matching identities do not prove

A matching digest identifies the same canonical input record, subject to the hash’s collision-resistance assumption. It does not prove that the algorithm is correct or that a new machine will reproduce every floating-point bit.

The source identity deliberately covers the forty first-course source files, not the evolving practical crate, this chapter’s reporting wrapper or interface. Toolchain, locked dependencies and arithmetic environment are recorded separately. A matching source digest is therefore not evidence that two execution environments match. The temporary checkpoint path is also outside the input identity; changing that path does not change the model’s fixed inputs.

Likewise, exact checkpoint reload is narrower than complete job resumption. This check does not restore every data cursor, in-flight operation or whole-job continuation state. No laptop throughput is measured here.

History: reporting conditions and measuring variation

A language-model score without its input conditions and scope does not identify what was evaluated or establish which capabilities were checked.

Model Cards for Model Reporting was presented in 2019. Mitchell and colleagues propose model documentation that includes intended uses and evaluation conditions; a model card is not itself proof of quality.

Model Cards describe evaluation conditions and intended uses; later fine-tuning experiments study variation across initializations and data orders. Reporting scope and measuring variation answer different questions.

In Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping, published in 2020, Dodge and colleagues study fine-tuning variability associated with initialization and data order for pretrained language models. This does not measure variability in the course’s tiny decoder.

Keep a fixed regression reference when changing an LLM pipeline, while recording separately the evidence needed for quality, seed sensitivity and hardware claims.

The Rust reporting contrast uses the same observation. Reading only the mean NLL with this expression gives a number:

trace
    .observation()
    .final_evaluation()
    .decoder()
    .mean_nll()

trace.report() also states the evaluation unit, input identity, replay accounting and limits. It does not retrain a historical model or execute a seed sweep.

    /// Read structured evidence from this one observation; never parse stdout
    /// or run another model to produce either reporting projection.
    pub fn report(&self) -> Result<String, ReferenceHandoffError> {
        let run = &self.observation;
        let evaluation = run.final_evaluation();
        let checkpoint = run.checkpoint();
        let generation = run.generation();
        let config = CapstoneConfig::tiny();
        let targets_per_schedule =
            config.updates() * config.update_batch_size() * config.context_length();
        let record = serde_json::json!({
            "schema_version": 1,
            "chapter": "01-reference-core-handoff",
            "reference_identity": self.digest.as_hex(),
            "source_revision": self.source_revision,
            "scope": "scalar-reference-fixed-fixture-regression",
            "documents": {
                "train": run.partitions().train_document_ids().len(),
                "validation": run.partitions().validation_document_ids().len(),
                "test": run.partitions().test_document_ids().len()
            },
            "model": { "parameters": run.training().parameter_count(), "context_tokens": config.context_length() },
            "training": {
                "target_count_basis": "derived-from-frozen-full-batch-schedules",
                "updates_per_schedule": config.updates(),
                "independent_schedules": 2,
                "valid_targets_per_schedule": targets_per_schedule,
                "valid_targets_primary_plus_replay": 2 * targets_per_schedule,
                "selected_step": run.training().selected_step(),
                "within_invocation_replay_bitwise": run.training().replay_bitwise()
            },
            "evaluation": {
                "unit": "overlapping-window-target-slot",
                "windows": evaluation.window_count(),
                "window_target_slots": evaluation.window_target_slot_count(),
                "document_transition_occurrences": evaluation.document_transition_occurrence_count(),
                "transition_multiplicity_counts": evaluation.transition_multiplicity_counts(),
                "decoder_mean_nll_nats_per_slot": evaluation.decoder().mean_nll(),
                "bigram_mean_nll_nats_per_slot": evaluation.bigram().mean_nll(),
                "once_per_transition_metric_reported": false,
                "independent_generalization_estimate": false
            },
            "checkpoint": {
                "bytes_roundtrip": checkpoint.bytes_roundtrip(),
                "model_bits_exact": checkpoint.model_bits_exact(),
                "optimizer_bits_exact": checkpoint.optimizer_bits_exact(),
                "tokenizer_exact": checkpoint.tokenizer_exact(),
                "logit_probe_text": checkpoint.logit_probe_text(),
                "logit_probe_ids": checkpoint.logit_probe_ids(),
                "probe_logits_bitwise": checkpoint.prompt_logits_bitwise(),
                "complete_job_resume_established": false
            },
            "generation": {
                "prompt": generation.prompt_text(),
                "token_ids": generation.generated_ids(),
                "decoded_text": generation.decoded_text(),
                "cached_reference_tokens_exact": generation.tokens_exact(),
                "cached_reference_decisions_bitwise": generation.decisions_bitwise(),
                "cached_reference_rng_exact": generation.rng_state_exact(),
                "useful_language_quality_established": false
            },
            "laptop_throughput_measured": false,
            "successor_chapter": SUCCESSOR_CHAPTER
        });
        let mut text = serde_json::to_string_pretty(&record)
            .map_err(|_| ReferenceHandoffError::Serialization)?;
        text.push('\n');
        Ok(text)
    }

Repeating seed $39$ checks repeatability under fixed conditions. It does not estimate variability across seeds. Those are complementary checks, not interchangeable evidence.

Run the reference without changing its inputs

The example obtains verified source evidence and owns its temporary checkpoint location. Its report comes directly from the checked scalar observation, not from parsing a previous human-readable report.

fn main() -> Result<(), Box<dyn Error>> {
    let source = verify_compiled_reference_source()?;
    let checkpoint = OwnedCheckpoint::new()?;
    let trace = run_reference_handoff(CORPUS, SPLITS, &checkpoint.file, &source)?;
    print!("{}", trace.report()?);
    Ok(())
}

Keep the three evidence layers separate

The digest identifies the fixed inputs. The counts and generated token IDs describe checked behavior on those inputs. The final card keeps future quality, job-resumption and hardware claims outside the reference’s evidence boundary.

Optional practice: inspect the boundary

From the repository root, with Docker and the Rust/site course image available, run the example and inspect its JSON report:

COURSE_CORPUS=false ./course run cargo run --quiet --offline --locked -p practical-ch01-reference-core-handoff

The output contains the fixed input identity and the checked scalar-reference observations described in this chapter.

1. In the JSON report printed by the Chapter 1 reference example, find the input identity, target-count basis and metric unit. Explain why a digest alone would not replace the replay and reload checks.

2. Run the pure identity-input mutation test. Identify its changes to configuration, corpus, split bytes and source identity without training. Explain why constructing an identity for changed inputs does not admit them to the fixed reference run.

   From the repository root, with Docker and the Rust/site course image available, run:

   COURSE_CORPUS=false ./course run cargo test --offline --locked -p practical-ch01-reference-core-handoff --test reference_handoff every_identity_input_is_bound_without_training -- --exact

   Expected result: one selected test passes. A zero-test result does not reproduce this check.

3. Explain why a successful checkpoint round trip cannot justify the statement “the complete training job can resume,” and why $1744$ must not be labeled “distinct test transitions.”

What the next extension must preserve

You now have an identified scalar reference and a precisely bounded report. A later change can be compared with that reference, while carrying its own input identity and evidence. Chapter 2 uses NVIDIA NeMo Curator to prepare supplied text, then loads the prepared documents through the practical crate’s Rust reader. New data, larger models, accelerator execution, quality evaluation and complete job resumption each require their own checks; none is already established by this handoff.