Corpus documents and frozen partitions

Give source documents their roles before learning

A tokenizer can learn which pieces of text to merge, and a model can learn weights that help it predict text. Measuring either system only on the observations used to fit it leaves an unanswered issue: the result may depend on those particular observations. It does not, by itself, establish how the system behaves on new text. We therefore decide which source text may fit the system and which text will serve other purposes before learning begins.

A corpus is a collection of source documents. A source document is one complete text whose start and end we preserve as a document boundary. In this chapter’s corpus, a complete scene in one language is one document. The single file rust/data/tiny-bilingual-corpus.json holds twelve JSON records, so the file itself is not the document unit. Each record has a language, a provenance group, and the text. Its document ID is the stable label used to name that record in the split; the label does not prove that its text is independent of other text.

A partition is the subset of corpus documents assigned to one role: training, validation, or test. A provenance group joins documents known to contain related source content. Here the English and Russian versions of the same scene share one group. They have different IDs and different text, but a translation in training can still expose content from its held-out partner. Whole documents and their known groups must therefore stay in one role.

Chapter 1 used a fixed mapping from text units to IDs. This chapter prepares the boundary needed when the mapping itself will be learned. The three roles separate fitting, choosing, and reporting:

Validation and test are holdout data because they stay outside fitting. They can still supply evaluation statistics, such as a measured error. Computing such a statistic is different from adding their pair counts to a tokenizer or using them to compute weight-update signals, called gradients. Validation influences the chosen system, so it is not untouched once we use it to choose. This chapter assigns the roles; it does not fit a tokenizer or model or compute a performance score.

The membership rule is easiest to see with a separate, six-document example. Let its complete corpus have IDs doc-01 through doc-06. The following lists are rejected:

train      = [doc-01, doc-02]
validation = [doc-03, doc-04]
test       = [doc-04, doc-05]

The lists contain six positions but only five distinct IDs. doc-04 occupies both a validation position and a test position; doc-06 occupies none. Their union, meaning all IDs found in any role, therefore omits doc-06. Their pairwise intersections, meaning IDs shared by two roles, expose the other failure: validation and test share doc-04, while training shares no ID with either role. Having the intended number of listed positions was not enough.

Keep training and validation as written and repair the test list:

train      = [doc-01, doc-02]
validation = [doc-03, doc-04]
test       = [doc-05, doc-06]

Now all six corpus IDs appear once. The union covers doc-01 through doc-06, and all three pairwise intersections are empty. This repair establishes complete, separate membership for the six named documents. It does not establish that their texts are unrelated or that they represent every kind of future text. The executable twelve-document corpus later in this lesson is a different example.

Express coverage and separation as sets

Treat the corpus and each role’s membership as sets of whole documents. The repaired example satisfies this statement:

The equality says that the three sets cover the corpus. The dotted union also requires that no document belongs to different roles. The intersection clause states that separation explicitly for any two distinct role labels. In the rejected example, validation and test shared doc-04, so that intersection was not empty; the missing doc-06 also broke coverage.

The manifest stores ordered ID lists, whereas the formula describes sets. A set forgets repeated positions within one list and forgets order altogether. Listing the same training ID twice cannot be justified by this formula, even if the set of training IDs looks unchanged. Reversing a list likewise leaves its set membership unchanged while changing its iteration order. The implementation checks those list properties separately. The formula also does not require nonempty roles, intact provenance groups, unchanged input bytes, or representative sampling.

Read each symbol in the split statement

These symbols describe membership and use, not a preferred split percentage. The appropriate amount of text for each role depends on the available corpus and the evaluation goal. The later eight/two/two counts in this chapter are document counts chosen for a small teaching example; they are not byte counts, token counts, or a universal ratio.

Preserve the source boundary behind each ID

Separating fitting from assessment predates modern LLMs. Stone’s 1974 paper studies cross-validatory choice and assessment. For language modeling, the unit that carries context also matters. Merity and colleagues introduced WikiText using article text to support longer dependencies. At larger scale, Brown and colleagues investigated training-data overlap with development and test benchmarks. These sources motivate attention to assessment and source context; they do not prescribe this course’s fixed bilingual split.

The existing Rust contrast shows one way a holdout label can conceal shared source context. It takes the single sentence north star glows softly and calls overlapping_word_windows("north star glows softly", 3). The two resulting windows are ["north", "star", "glows"] and ["star", "glows", "softly"]. shared_words reports ["star", "glows"] in the first window’s order:

pub fn overlapping_word_windows(text: &str, width: usize) -> Vec<Vec<&str>> {
    let words = text.split_whitespace().collect::<Vec<_>>();
    if width == 0 || width > words.len() {
        return Vec::new();
    }
    words.windows(width).map(<[&str]>::to_vec).collect()
}

/// Lists tokens shared by two separately named excerpts in left-hand order.
pub fn shared_words<'a>(left: &'a [&'a str], right: &[&str]) -> Vec<&'a str> {
    left.iter()
        .copied()
        .filter(|word| right.contains(word))
        .collect()
}

If we name those windows separately and put one in training and the other in validation, the IDs differ but source context crosses the role boundary. The problem is their derivation from the same sentence. Two unrelated documents using a common word do not, for that reason alone, demonstrate this leakage. Here, data leakage means that information available to fitting crosses a boundary meant to keep assessment separate from that fitting.

Assign whole source documents and known provenance groups first. Fit tokenizer statistics only on training documents, freeze the learned rules, and apply those rules separately to text in each role. Any later token windows must remain inside an original document and retain its role. Renaming excerpts cannot replace that order. This prevents the demonstrated overlap path; it is not a claim that every historical experiment leaked or that every overlap improves every score.

Inspect the existing document and split checks

The existing loader keeps the source units used by the explanation. Its input is JSON text as &str, so valid UTF-8 is already guaranteed at that boundary. serde_json::from_str handles JSON syntax and typed fields. Corpus::from_json separately validates document metadata and nonempty text, rejects duplicate document IDs and exact duplicate decoded texts, and preserves the array’s source order:

    pub fn from_json(source: &str) -> Result<Self, CorpusError> {
        let decoded: Vec<DocumentJson> = serde_json::from_str(source)
            .map_err(|error| CorpusError::new(format!("invalid corpus JSON: {error}")))?;
        if decoded.is_empty() {
            return Err(CorpusError::new("corpus contains no documents"));
        }

        let mut documents = Vec::with_capacity(decoded.len());
        for (index, document) in decoded.into_iter().enumerate() {
            let position = index + 1;
            for (value, label) in [
                (&document.id, "document ID"),
                (&document.language, "language"),
                (&document.provenance_group, "provenance group"),
            ] {
                if !is_kebab_identifier(value) {
                    return Err(CorpusError::new(format!(
                        "corpus document {position} {label} must be lowercase ASCII kebab case"
                    )));
                }
            }
            if document.text.trim().is_empty() {
                return Err(CorpusError::new(format!(
                    "corpus document {position} text is empty"
                )));
            }
            if documents
                .iter()
                .any(|existing: &Document| existing.id == document.id)
            {
                return Err(CorpusError::new(format!(
                    "duplicate document ID {}",
                    document.id
                )));
            }
            if documents
                .iter()
                .any(|existing: &Document| existing.text == document.text)
            {
                return Err(CorpusError::new(
                    "duplicate document text would leak identical content",
                ));
            }
            documents.push(Document {
                id: document.id,
                language: document.language,
                provenance_group: document.provenance_group,
                text: document.text,
            });
        }

        Ok(Self {
            documents,
            checksum: format!("fnv1a64:{:016x}", fnv1a64(source.as_bytes())),
        })
    }

A split manifest records the fixed role assignments. The checked-in rust/data/splits.json names schema version 1, strategy fixed-paired-document-holdout-v1, and checksum fnv1a64:723b071980ae8a22. SplitManifest::from_json decodes its fields; successful JSON decoding alone says nothing about whether its assignments are a valid split of this corpus.

The separate partition method checks the supported schema and strategy and compares the recorded checksum with the checksum recomputed from the corpus JSON text’s UTF-8 bytes. It also requires known IDs, no ID occupying more than one position within or across lists, complete coverage, nonempty roles, source order within each role, and declared provenance groups that stay together. Only after those checks succeed does it return borrowed document views:

    pub fn partition<'a>(&self, corpus: &'a Corpus) -> Result<CorpusPartitions<'a>, CorpusError> {
        if self.schema_version != SPLIT_SCHEMA_VERSION {
            return Err(CorpusError::new(format!(
                "unsupported split schema version {}",
                self.schema_version
            )));
        }
        if self.strategy != SPLIT_STRATEGY {
            return Err(CorpusError::new(format!(
                "unsupported split strategy {}",
                self.strategy
            )));
        }
        if self.corpus_checksum != corpus.checksum {
            return Err(CorpusError::new(format!(
                "corpus checksum mismatch: manifest={}, actual={}",
                self.corpus_checksum, corpus.checksum
            )));
        }
        for partition in [Partition::Train, Partition::Validation, Partition::Test] {
            if self.ids(partition).is_empty() {
                return Err(CorpusError::new(format!(
                    "{} partition is empty",
                    partition.label()
                )));
            }
            validate_source_order(corpus, partition, self.ids(partition))?;
        }

        let mut seen = Vec::new();
        for partition in [Partition::Train, Partition::Validation, Partition::Test] {
            for id in self.ids(partition) {
                if corpus.document(id).is_none() {
                    return Err(CorpusError::new(format!(
                        "{} partition contains unknown document {id}",
                        partition.label()
                    )));
                }
                if seen.contains(&id) {
                    return Err(CorpusError::new(format!(
                        "document {id} appears in more than one manifest position"
                    )));
                }
                seen.push(id);
            }
        }
        if seen.len() != corpus.documents.len() {
            let missing = corpus
                .documents
                .iter()
                .find(|document| !seen.iter().any(|id| id.as_str() == document.id))
                .map_or("<unknown>", Document::id);
            return Err(CorpusError::new(format!(
                "manifest does not cover corpus document {missing}"
            )));
        }

        for document in &corpus.documents {
            let assigned = self.assignment(document.id()).ok_or_else(|| {
                CorpusError::new(format!(
                    "manifest does not cover corpus document {}",
                    document.id
                ))
            })?;
            if let Some(related) = corpus.documents.iter().find(|candidate| {
                candidate.provenance_group == document.provenance_group
                    && self.assignment(candidate.id()) != Some(assigned)
            }) {
                return Err(CorpusError::new(format!(
                    "provenance group {} is split between {} and {}",
                    document.provenance_group, document.id, related.id
                )));
            }
        }

        let mut partitions = CorpusPartitions {
            train: Vec::new(),
            validation: Vec::new(),
            test: Vec::new(),
        };
        for document in &corpus.documents {
            match self.assignment(document.id()).ok_or_else(|| {
                CorpusError::new(format!(
                    "manifest does not cover corpus document {}",
                    document.id
                ))
            })? {
                Partition::Train => partitions.train.push(document),
                Partition::Validation => partitions.validation.push(document),
                Partition::Test => partitions.test.push(document),
            }
        }
        Ok(partitions)
    }

The fixed manifest makes assignments repeatable, and retained corpus order makes later iteration and printed evidence repeatable. Neither property establishes statistical independence. Exact duplicate rejection does not detect every near-duplicate or copied passage; known group labels do not discover every unrecognized relationship. A deterministic split may still sample a narrow topic range or fail to represent text encountered later.

The 64-bit FNV-1a value is a limited drift check. A mismatch reports a different checksum. A match reports only that recomputation produced the recorded value: collisions are possible, so it does not prove byte identity, authorship, licensing, or cryptographic authenticity. The corpus README records the teaching text’s origin separately.

The executable prints the accepted assignments, the window overlap, and the training-document count for the next chapter:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let corpus = Corpus::from_json(CORPUS_JSON)?;
    let manifest = SplitManifest::from_json(SPLIT_MANIFEST)?;
    let partitions = manifest.partition(&corpus)?;

    println!("corpus checksum: {}", corpus.checksum());
    println!("documents: {}", corpus.documents().len());
    print_partition("train", &partitions, Partition::Train);
    print_partition("validation", &partitions, Partition::Validation);
    print_partition("test", &partitions, Partition::Test);
    println!("complete: yes");
    println!("disjoint: yes");
    println!("provenance groups intact: yes");

    let excerpts = overlapping_word_windows("north star glows softly", 3);
    println!("historical excerpt A: {:?}", excerpts[0]);
    println!("historical excerpt B: {:?}", excerpts[1]);
    println!(
        "shared context: {:?}",
        shared_words(&excerpts[0], &excerpts[1])
    );
    println!("safe split unit: whole source document");
    println!(
        "chapter 3 tokenizer input: train only ({} documents)",
        partitions.training_documents().len()
    );
    println!(
        "held out: validation={} test={}",
        partitions.documents(Partition::Validation).len(),
        partitions.documents(Partition::Test).len()
    );

    Ok(())
}

These views make the intended choice of training input explicit. They do not prevent every other caller from reading the source file or prove that differently named documents contain independent text.

Follow all twelve documents in the fixed corpus

The corpus represents six scenes in English and Russian. Each scene contributes two documents to one pair-* provenance group. Training contains, in source order, the river-dawn, clock-shop, rain-library, and bee-garden pairs: eight documents in four groups. Validation contains the night-station pair: two documents in one group. Test contains the winter-window pair: two documents in one group.

The figure lists every exact ID, language, and group. Its three lists are peer sets, not stages through which a document moves. Read each card as one whole document and read the cards within each role in corpus source order:

For example, en-night-station has language en and ru-night-station has language ru; both have group pair-night-station and both belong to validation. Different languages do not make the shared scene independent. The same pairing rule holds for pair-river-dawn, pair-clock-shop, pair-rain-library, pair-bee-garden, and pair-winter-window. Every corpus ID appears in exactly one role, with eight training documents and two documents in each holdout role.

This is an original, public teaching corpus small enough to inspect. It supports repeatable examples, not a claim of useful real-world language-model quality. Rerunning a known test result checks for regression; it does not supply a fresh, independent estimate of generalization each time.

Optional inspection and reproduction

cargo test --locked -p llm-from-scratch --lib corpus::tests::
cargo test --locked -p ch02-corpus-partitions
cargo run --quiet --locked -p ch02-corpus-partitions | diff -u rust/demos/ch02-corpus-partitions/expected.txt -

Give BPE only the training documents

Chapter 3’s BPE trainer uses partitions.training_documents(): the eight whole training documents. It starts a separate byte-token sequence for each document and counts adjacent pairs inside that sequence. The last byte of one document does not become adjacent to the first byte of the next. Validation and test supply no pair counts or learned merge rules.

After fitting, the learned rules can be frozen and applied to held-out text without learning from it. Later token windows must also stay within original document boundaries and keep the document’s role. This preserves both boundaries: which text fits learned state, and which text belongs to one source.

The same responsibilities continue into model training: training fits weights, validation guides choices, and test reports after selection. Chapter 34 later demonstrates that order with a local evaluator inside one execution. Its gate cannot enforce unique access across all callers or later executions, and its caller-supplied provenance labels do not independently establish data history. The later deliberately chosen, known comparison remains regression evidence, not an untouched generalization estimate. The boundary introduced here makes those later claims precise without pretending that a small fixed corpus proves more than it does.