Corpus documents and frozen partitions

Audit IDs before looking at text

Suppose a six-document corpus has the stable IDs doc-01 through doc-06. Before reading further, find the faults in this tempting split:

train      = [doc-01, doc-02]
validation = [doc-03, doc-04]
test       = [doc-04, doc-05]

There are two independent failures. doc-04 belongs to both validation and test, so those sets intersect. doc-06 belongs to no set, so their union does not cover the corpus. One repair is:

train      = [doc-01, doc-02]
validation = [doc-03, doc-04]
test       = [doc-05, doc-06]

The repaired lists are safe only if every ID in them names one complete source document. Different IDs alone do not prove that the underlying text is different. For example, imagine cutting north star glows softly into two windows: call them window-A (north star glows) and window-B (star glows softly). The windows have different names, but both contain star glows. If the first window is placed in training and the second in validation, the two partitions share source text and context.

The safe order is therefore: assign the original whole document first. Place it in exactly one partition, tokenize it, and keep every window created from it in that same partition. Never create overlapping windows first and distribute them independently, because distinct window IDs would hide the shared source context.

The repository fixture contains twelve short UTF-8 documents: English and Russian versions of six scenes. A translated pair shares one provenance group and must stay together. Four pairs supply eight training documents; one pair supplies two validation documents; the final pair supplies two test documents. These are small, auditable fixture counts—not a universal recommendation for split ratios.

Coverage and separation in one statement

The split’s set invariant is:

The dotted union says two things at once: the three roles cover the entire corpus, and their membership does not overlap. The intersection clause repeats the second promise explicitly for every pair of distinct roles. Only the training set may teach BPE merges or model parameters. Validation may later guide choices. In the demonstrated course execution, test cannot fit or select: Chapter 34 gives one local evaluator instance access only after selection. That order does not claim that the checked-in fixture has never been read during repository development.

This notation is necessary but not sufficient for the implementation. It says nothing about empty roles, byte drift, unknown IDs, source order, provenance pairs, or whether the sample represents a broader population. The implementation checks those operational properties separately.

Symbol glossary

Notice what is absent: there is no required percentage in the formula. The appropriate ratio depends on data volume and evaluation goals. Here, exact membership is the invariant. The 8 / 2 / 2 (eight documents in training, two in validation, and two in test) assignment is specific to this teaching example.

Key idea: assign whole source documents before creating token windows; otherwise new IDs can hide shared context across partitions.

Before dependable holdout boundaries

Evaluating on fitted text was historically convenient, but a training score answers how well a system fits observations it already saw. It does not provide independent evidence about new text. Cross-validation and fixed holdouts developed to separate fitting from predictive assessment; language modeling adds the subtle question of where the underlying source boundary lies.

A common-looking but unsafe sequence is: create many partly overlapping excerpts, give each excerpt a fresh ID, then randomly distribute those IDs. The runnable contrast starts from north star glows softly and makes two three-word windows. They have different positions and names, yet both contain star glows:

pub fn overlapping_word_windows(text: &str, width: usize) -> Vec<Vec<&str>> {
    let words = text.split_whitespace().collect::<Vec<_>>();
    if width == 0 || width > words.len() {
        return Vec::new();
    }
    words.windows(width).map(<[&str]>::to_vec).collect()
}

/// Lists tokens shared by two separately named excerpts in left-hand order.
pub fn shared_words<'a>(left: &'a [&'a str], right: &[&str]) -> Vec<&'a str> {
    left.iter()
        .copied()
        .filter(|word| right.contains(word))
        .collect()
}

For the historical trail, Stone’s 1974 paper formalizes cross-validatory choice and assessment. Merity et al. later introduce the long-context WikiText language-modeling corpus, while Brown et al. include a dedicated training-data contamination analysis for a large language model.

Whole-document assignment prevents that particular leakage path. It does not prove that the corpus is representative, nor does an ID detect every copied or near-duplicate passage. Validation also is not untouched once it guides a decision. The generic test role is post-selection evidence. In this course, the enforceable guarantee is narrower: test cannot affect the selected state inside one execution, while the checked-in result may be rerun as repository regression evidence.

Carry forward: a clean holdout boundary makes later tokenizer and model measurements interpretable, but it does not make a small corpus representative by itself.

Enforce the boundary in Rust

The corpus file is an ordinary JSON array, with one complete source document in each item. Corpus::from_json accepts the JSON text as &str, so Rust guarantees valid UTF-8 before the method begins. It passes that text to serde_json::from_str, which parses the JSON and deserializes each item’s four required fields into a private Rust record. The method then checks the rules that define a valid document for this example: the ID, language, and provenance group begin with a lowercase ASCII letter and otherwise contain only lowercase ASCII letters or digits in nonempty hyphen-separated segments; the text contains at least one non-whitespace character; IDs and decoded texts do not repeat; and array order is preserved:

    pub fn from_json(source: &str) -> Result<Self, CorpusError> {
        let decoded: Vec<DocumentJson> = serde_json::from_str(source)
            .map_err(|error| CorpusError::new(format!("invalid corpus JSON: {error}")))?;
        if decoded.is_empty() {
            return Err(CorpusError::new("corpus contains no documents"));
        }

        let mut documents = Vec::with_capacity(decoded.len());
        for (index, document) in decoded.into_iter().enumerate() {
            let position = index + 1;
            for (value, label) in [
                (&document.id, "document ID"),
                (&document.language, "language"),
                (&document.provenance_group, "provenance group"),
            ] {
                if !is_kebab_identifier(value) {
                    return Err(CorpusError::new(format!(
                        "corpus document {position} {label} must be lowercase ASCII kebab case"
                    )));
                }
            }
            if document.text.trim().is_empty() {
                return Err(CorpusError::new(format!(
                    "corpus document {position} text is empty"
                )));
            }
            if documents
                .iter()
                .any(|existing: &Document| existing.id == document.id)
            {
                return Err(CorpusError::new(format!(
                    "duplicate document ID {}",
                    document.id
                )));
            }
            if documents
                .iter()
                .any(|existing: &Document| existing.text == document.text)
            {
                return Err(CorpusError::new(
                    "duplicate document text would leak identical content",
                ));
            }
            documents.push(Document {
                id: document.id,
                language: document.language,
                provenance_group: document.provenance_group,
                text: document.text,
            });
        }

        Ok(Self {
            documents,
            checksum: format!("fnv1a64:{:016x}", fnv1a64(source.as_bytes())),
        })
    }

The loader computes a deterministic 64-bit FNV-1a checksum of the UTF-8 bytes of the supplied JSON string. include_str! preserves the checked-in file’s exact text, so the canonical checksum still detects the accidental source mutations exercised by the tests. A match only reports that recomputation produced the recorded 64-bit value; because FNV collisions exist, it is not proof of byte identity, authorship, licensing, or trustworthy provenance.

SplitManifest::from_json uses a separate private Rust record for the manifest’s six required fields and the same &str boundary. For both inputs, serde_json::from_str handles JSON syntax and typed field deserialization. The derived records require every declared field and its value type, reject duplicate fields, and use deny_unknown_fields to reject extra fields. Those format checks do not validate whether the document assignments satisfy the train/validation/test invariants.

partition performs the split-specific checks. It rejects an unsupported schema version or strategy, checksum drift, unknown or repeated IDs, omissions, empty roles, reordered IDs, and a provenance group split between roles. Only after every check succeeds does it return borrowed document views:

    pub fn partition<'a>(&self, corpus: &'a Corpus) -> Result<CorpusPartitions<'a>, CorpusError> {
        if self.schema_version != SPLIT_SCHEMA_VERSION {
            return Err(CorpusError::new(format!(
                "unsupported split schema version {}",
                self.schema_version
            )));
        }
        if self.strategy != SPLIT_STRATEGY {
            return Err(CorpusError::new(format!(
                "unsupported split strategy {}",
                self.strategy
            )));
        }
        if self.corpus_checksum != corpus.checksum {
            return Err(CorpusError::new(format!(
                "corpus checksum mismatch: manifest={}, actual={}",
                self.corpus_checksum, corpus.checksum
            )));
        }
        for partition in [Partition::Train, Partition::Validation, Partition::Test] {
            if self.ids(partition).is_empty() {
                return Err(CorpusError::new(format!(
                    "{} partition is empty",
                    partition.label()
                )));
            }
            validate_source_order(corpus, partition, self.ids(partition))?;
        }

        let mut seen = Vec::new();
        for partition in [Partition::Train, Partition::Validation, Partition::Test] {
            for id in self.ids(partition) {
                if corpus.document(id).is_none() {
                    return Err(CorpusError::new(format!(
                        "{} partition contains unknown document {id}",
                        partition.label()
                    )));
                }
                if seen.contains(&id) {
                    return Err(CorpusError::new(format!(
                        "document {id} appears in more than one manifest position"
                    )));
                }
                seen.push(id);
            }
        }
        if seen.len() != corpus.documents.len() {
            let missing = corpus
                .documents
                .iter()
                .find(|document| !seen.iter().any(|id| id.as_str() == document.id))
                .map_or("<unknown>", Document::id);
            return Err(CorpusError::new(format!(
                "manifest does not cover corpus document {missing}"
            )));
        }

        for document in &corpus.documents {
            let assigned = self.assignment(document.id()).ok_or_else(|| {
                CorpusError::new(format!(
                    "manifest does not cover corpus document {}",
                    document.id
                ))
            })?;
            if let Some(related) = corpus.documents.iter().find(|candidate| {
                candidate.provenance_group == document.provenance_group
                    && self.assignment(candidate.id()) != Some(assigned)
            }) {
                return Err(CorpusError::new(format!(
                    "provenance group {} is split between {} and {}",
                    document.provenance_group, document.id, related.id
                )));
            }
        }

        let mut partitions = CorpusPartitions {
            train: Vec::new(),
            validation: Vec::new(),
            test: Vec::new(),
        };
        for document in &corpus.documents {
            match self.assignment(document.id()).ok_or_else(|| {
                CorpusError::new(format!(
                    "manifest does not cover corpus document {}",
                    document.id
                ))
            })? {
                Partition::Train => partitions.train.push(document),
                Partition::Validation => partitions.validation.push(document),
                Partition::Test => partitions.test.push(document),
            }
        }
        Ok(partitions)
    }

The executable joins the canonical corpus and manifest, prints every assignment, then uses training_documents() for the Chapter 3 handoff:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let corpus = Corpus::from_json(CORPUS_JSON)?;
    let manifest = SplitManifest::from_json(SPLIT_MANIFEST)?;
    let partitions = manifest.partition(&corpus)?;

    println!("corpus checksum: {}", corpus.checksum());
    println!("documents: {}", corpus.documents().len());
    print_partition("train", &partitions, Partition::Train);
    print_partition("validation", &partitions, Partition::Validation);
    print_partition("test", &partitions, Partition::Test);
    println!("complete: yes");
    println!("disjoint: yes");
    println!("provenance groups intact: yes");

    let excerpts = overlapping_word_windows("north star glows softly", 3);
    println!("historical excerpt A: {:?}", excerpts[0]);
    println!("historical excerpt B: {:?}", excerpts[1]);
    println!(
        "shared context: {:?}",
        shared_words(&excerpts[0], &excerpts[1])
    );
    println!("safe split unit: whole source document");
    println!(
        "chapter 3 tokenizer input: train only ({} documents)",
        partitions.training_documents().len()
    );
    println!(
        "held out: validation={} test={}",
        partitions.documents(Partition::Validation).len(),
        partitions.documents(Partition::Test).len()
    );

    Ok(())
}

Run the tests before comparing the checked output:

cargo test --workspace --locked
cargo run --quiet --locked -p ch02-corpus-partitions | diff -u rust/demos/ch02-corpus-partitions/expected.txt -

The focused tests include a successful three-way assignment and independent mutations for malformed JSON, checksum drift, missing, repeated, unknown, or reordered IDs, empty roles, and split provenance. A successful diff prints nothing.

Implementation takeaway: validation happens before a training view is exposed, so Chapter 3 cannot accidentally learn from held-out documents.

Verify all twelve assignments at a glance

The three regions below are peer sets, not steps in a left-to-right pipeline. Read each card as one indivisible source document. Use each region heading to identify its partition, then use the stable ID on each card to verify that every document appears exactly once. There should be eight training cards, two validation cards, and two test cards. English and Russian partners with the same provenance label must appear in the same region.

The diagram shows the same accepted corpus and splits.json assignment as the executable. Every corpus ID appears exactly once in source order, and every provenance group remains within one role. On a narrow screen the regions stack, but membership and source order remain unchanged.

Predict, then validate

1. For the rejected six-ID split, write the union and the three pairwise intersections before repairing it.
2. Decide which role may count BPE pairs, which may choose a checkpoint, and which may supply the final reported loss.
3. Explain why two separately named overlapping excerpts can still leak context.
4. Predict the first error after adding doc-07 to the corpus without changing the manifest. What error would remain if you refreshed only its recorded checksum?
5. Explain why reversing a role’s IDs preserves set membership but breaks the reproducible ordering contract.
6. State what a matching FNV value reports and why it is not proof of byte identity.
7. Name one risk that a valid deterministic document split still cannot eliminate.

Change a copied manifest only after the fixed case passes. For each mutation, predict which invariant rejects it before running the test.

Hand only training documents to BPE

Chapter 3 receives partitions.training_documents(): eight borrowed whole documents and no validation or test view. Its adjacent-byte pair counter will reset at every document boundary, so neither a pair nor a later training window may bridge two sources. Holdout bytes contribute no learned vocabulary or merge rank.

Each candidate tokenizer or model configuration fits its normalization rules, merge statistics, vocabulary, and weights on training documents. Validation may compare those training-fitted candidates and select a checkpoint; it contributes no fitting counts or gradients. Test neither fits nor selects. In the demonstrated execution, the local evaluator receives it only after validation selection; repository reuse of the known fixture is regression evidence rather than a new independent estimate. This role boundary survives every later chapter, from BPE through the final decoder comparison.