Prepare text externally, then load it in Rust

The problem: the model learns from the text we let through

A language model learns to predict tokens from the documents supplied during training. Its optimizer does not decide whether those documents are appropriate for the task, whether we may use them, or whether a purportedly unseen test example also appears in the training input. Those decisions must happen before training.

Repeated text creates another problem. If one training pass visits every stored document occurrence, each copy of a document supplies another copy of its token targets. The repeated text receives more influence without adding new examples. Removing duplicates changes that weighting; it does not merely save disk space.

Corpus preparation therefore has two outputs: selected text and evidence about how it was selected. We need source context, declared exclusion rules, duplicate handling and protected evaluation boundaries. A large file or a successful download supplies none of those conclusions on its own.

We will use NVIDIA NeMo Curator to prepare text. It is an external tool, not part of the Rust language model, and another preparation tool can replace it if it produces the same documented input for the model. Rust will only load the prepared documents. The extended tokenizer in Chapter 3 will then learn from the training documents, not the validation or test documents.

The solution: prepare documents, then load a stable format

Keep one document as one record throughout the preparation pipeline. Retain a stable document ID and the source information needed to trace that record. A source record should identify where the text came from, its revision and the applicable license and attribution. A checksum identifies selected bytes; it does not establish permission to use those bytes or tell us what they contain.

Apply declared quality rules, inspect what those rules exclude, and handle duplicates before final training input is frozen. Quality rules are proxies: for example, a minimum word count rejects short records, not every unsuitable record. A rule can also remove useful text. Inspect both survivors and exclusions rather than interpreting a low exclusion rate as success.

Keep privacy review separate from quality scoring. Passing a word-count filter or removing duplicates does not remove names, addresses, credentials or other sensitive content. Withhold unresolved records from training; do not copy their sensitive text into public reports. Retain enough source and transformation information to identify affected derived corpora if an input is withdrawn. Removing a source from a later corpus does not erase what an already trained model may have learned from it.

A six-document example

The practical example uses course-authored text, not a result for a large training corpus. Its JSONL input contains these six documents:

Configure NeMo’s word-count filter to retain records with at least $3$ and at most $20$ words. Its English setting counts whitespace-separated items; that proxy is not a linguistic definition of a word. The $4$ story occurrences survive; hi and the punctuation-only record do not. Exact duplicate removal then leaves $3$ distinct story texts. The punctuation-only record has one whitespace-separated item too; this particular rule rejects it for length, not because it recognizes punctuation as unsuitable language-model training text. The duplicate-removal workflow may retain either cat-story ID; the result we care about here is one cat-story occurrence, not a promised representative ID or output-file order.

Before deduplication, the cat story occupies $2$ of the $4$ surviving document occurrences. After deduplication, it occupies $1$ of $3$ distinct documents. Under uniform sampling of these whole-stage document occurrences before separating their roles, its selection probability would change from $2/4$ to $1/3$. That calculation illustrates duplicate weighting, not a sampler used to train on this fixture. Actual token-target weighting also depends on document length, split selection and the batching policy; document fractions are not token fractions.

NeMo’s exact method groups hashes of text and identifies copies to remove. It does not identify paraphrases, and hash grouping is not an adversarial collision-proof comparison. NeMo’s fuzzy method uses approximate similarity search for near-duplicates. Its configuration trades missed matches against unwanted matches; it does not certify a universal similarity cutoff or detect all ways an evaluation example could have leaked into training.

Freeze the roles of the prepared documents

The training split supplies examples for fitting the tokenizer and model. The validation split supplies examples for selecting settings or deciding when to stop training. The test split is reserved for final evaluation. Do not move a difficult held-out document into training after seeing its score.

Related records must not be scattered across those roles. Treat an identified duplicate family or other known related source group as one unit when assigning splits. Otherwise two different IDs can put substantially the same example in both training and evaluation. For this fixture, the two cat-story copies share family-a; their different IDs do not make them independent examples.

For the exercise, a supplied policy assigns family-a, family-e and family-f to training, family-b to validation and family-c to test. We fix that policy before running the filters. Both cat-story copies therefore have the same role regardless of which ID duplicate removal keeps. The prepared fixture has one document in each role: the cat story in training, the dog story in validation and the rain story in test.

The recipe uses Pandas to join those declared assignments to the source-group metadata. It refuses an unassigned group rather than choosing a convenient role after seeing the output. This checks the supplied groups, not whether we have discovered every related pair. One held-out story is also far too little evidence for a meaningful model-quality estimate.

Split disjointness and decontamination are different checks. Disjoint IDs do not prove different text. Removing exact duplicates does not prove that near-copies or external benchmark examples are absent. Declare the protected evaluation material and the overlap method before inspecting results, report its measured scope, and keep unresolved overlaps out of training. A check against a named evaluation set does not establish internet-wide contamination freedom.

Export documents, not preparation-specific objects

The handoff to Rust is UTF-8 JSON Lines: one JSON object per physical line. Each object has a nonblank string id and a nonblank string text. Other fields are retained as metadata. For example:

{"id":"story-c","text":"A dog runs.","source_group":"family-b"}

The document ID is a label, not a token ID or a required filename. The text is still text; tokenization comes next. A newline inside document text must be escaped as \n inside the JSON string, so it does not create a second record. Source and split records remain linked to the exported documents, but a JSONL file alone does not certify that those decisions were made correctly.

Formula: distinguish retention from quality

For a nonempty input corpus, the final document-retention fraction is

Here $N_{\mathrm{input}}$ is the number of input document occurrences, $N_{\mathrm{prepared}}$ is the number remaining after the declared preparation stages, and $r_{\mathrm{keep}}$ is a dimensionless fraction. Neither count is a byte count or token count. With zero input documents, the fraction is undefined; report that condition rather than inventing a zero-quality score.

In the six-document example, quality filtering retains $4/6$ of the original input. Duplicate removal retains $3/4$ of its own input: the four records that reached that stage. Their product is the final retention fraction,

Changing the denominator changes the question. Reporting $1/6$ as the duplicate stage’s exclusion fraction would use the original input population rather than the $4$ records the duplicate stage examined. Keep every stage’s entering and leaving document counts so its fraction can be interpreted locally.

Keeping half the input says how much remains under this policy. It does not say that half was bad, that the survivors are safe, or that a model trained on them will perform well. Those claims need different evidence.

Practical preparation with NeMo Curator

This workflow uses a separately built NeMo Curator image. You need Linux, Docker, a compatible NVIDIA GPU and driver, and the NVIDIA Container Toolkit so Docker can expose the GPU. The pinned NeMo release must be compatible with that CUDA/driver stack. Rust’s prepared-corpus reader does not require a GPU or CUDA.

Run the following commands from the repository root. The image build may use network access to obtain the external tool. It does not acquire a training corpus or add NeMo to the Rust/site image:

docker build --file docker/nemo-curator.Dockerfile \
  --tag learn-llm-nemo-curator:26.07 .

With the NeMo image built and the NVIDIA GPU available to Docker, run from the repository root. Create a fresh output directory for the exercise, then run the tool with network access disabled. The read-only mount supplies the recipe and its six-record fixture; the writable mount receives generated preparation output:

mkdir -p .build/corpus/practical-ch02-exact-fixture
docker run --rm --network none --gpus all --memory=8g --cpus=2 --shm-size=1g \
  --hostname course-nemo --add-host course-nemo:127.0.0.1 \
  --mount type=bind,source="$PWD/tools/nemo-curator",target=/course,readonly \
  --mount type=bind,source="$PWD/.build/corpus/practical-ch02-exact-fixture",target=/output \
  learn-llm-nemo-curator:26.07 \
  /course/prepare_corpus.py \
    --input /course/fixtures/raw.jsonl --output /output \
    --split-policy /course/fixtures/splits.json

Use an empty output directory. If the directory already contains an earlier attempt, choose a different directory rather than overwriting its evidence. The hostname mapping lets the local Ray workers resolve their container on loopback; it does not enable external network access.

The recipe invokes NeMo’s reader, configured word-count filter, exact duplicate identification and duplicate-removal workflow. Identification produces IDs to remove; it is not already the prepared text. The removal workflow must consume those IDs from the same input and ID policy, then export the remaining documents.

The filter configuration in tools/nemo-curator/prepare_corpus.py uses the library’s pipeline and stages; the recipe does not implement its own filter:

Pipeline(
    name="course_corpus_quality",
    stages=[
        JsonlReader(file_paths=str(input_path), read_kwargs=read_kwargs),
        ScoreFilter(
            filter_obj=WordCountFilter(min_words=min_words, max_words=max_words, lang="en"),
            score_field="word_count",
        ),
        JsonlWriter(path=str(quality_path), mode="error"),
    ],
).run(executor=executor)

The surrounding recipe supplies the reader settings, starts a local Ray client, and runs NeMo’s exact-identification and text-removal workflows. The defaults set min_words to 3 and max_words to 20. Command-line options change those values; retain them with the output report when comparing different policies. The supplied split-policy file is independent of the observed scores.

Inspect the generated report and exported files with these commands, still from the repository root:

cat .build/corpus/practical-ch02-exact-fixture/report.json
cat .build/corpus/practical-ch02-exact-fixture/prepared.jsonl
cat .build/corpus/practical-ch02-exact-fixture/train.jsonl

For this fixture, the expected stage counts are $6$ input documents, $4$ after word-count filtering and $3$ after exact duplicate removal. The final texts are A cat naps., A dog runs. and Rain feeds trees.; do not require one particular cat-story ID. The report also records one document in each role. prepared.jsonl contains all three documents for inspection; train.jsonl, validation.jsonl and test.jsonl contain their respective selections and retain the split metadata.

This is a tiny execution example with a fixed assignment of known source groups. It is not an approval of a larger corpus, a privacy check, proof that all related groups were found or a trained-model result. The helper limits input to $4$ MiB and $4096$ documents by default so the example is inspectable; those are not a recipe for processing the previously acquired large corpus. A bulk run needs its own feasible configuration and complete selection evidence.

For a real corpus, also retain input provenance, the tool version and configuration, excluded-document counts, the duplicate policy, split assignments and the scope of overlap and privacy review. A successful small fixture does not establish that the same resource settings fit a larger dataset. Select and measure that workload separately, preserving the raw inputs and every failed attempt.

Load the prepared output in Rust

PreparedCorpusReader accepts any caller-supplied BufRead stream. A file, memory buffer or database export can supply that stream without changing the loader. Standard buffered I/O reads physical lines; Serde JSON parses their objects. The loader preserves decoded text and metadata. It does not normalize text, reject duplicate content or assign a split.

        let document: PreparedDocument = serde_json::from_slice(&self.buffer)
            .map_err(|source| CorpusReadError::InvalidJson { line, source })?;
        for (field, value) in [("id", &document.id), ("text", &document.text)] {
            if value.trim().is_empty() {
                return Err(CorpusReadError::EmptyField { line, field });
            }
        }

The caller chooses maximum source-record bytes, document count and cumulative decoded-text bytes. Record bytes include the physical line ending; decoded text bytes count the UTF-8 representation of the text value, not JSON syntax or metadata. An oversized record, malformed JSON, missing required field, failed read or exceeded limit stops the stream with an error. No failed document is returned or counted. Documents returned earlier are not rolled back.

    fn next(&mut self) -> Option<Self::Item> {
        if self.finished {
            return None;
        }
        match self.read_document() {
            Ok(Some(document)) => Some(Ok(document)),
            Ok(None) => {
                self.finished = true;
                None
            }
            Err(error) => {
                self.finished = true;
                Some(Err(error))
            }
        }
    }

The demo reads standard input, retains only the first three document descriptions for its summary, and emits that summary only after the complete input succeeds. It can therefore inspect a large stream without retaining every text. This is not a transactional storage adapter: a caller storing documents as they arrive must provide its own staging if partial results must stay unpublished.

pub fn summarize<R: BufRead>(
    source: R,
    limits: ReadLimits,
) -> Result<CorpusSummary, CorpusReadError> {
    let mut documents = PreparedCorpusReader::new(source, limits)?;
    let mut first_documents = Vec::with_capacity(SUMMARY_DOCUMENTS);
    for document in documents.by_ref() {
        let document = document?;
        if first_documents.len() < SUMMARY_DOCUMENTS {
            first_documents.push(DocumentSummary {
                id: document.id,
                metadata_fields: document.metadata.keys().cloned().collect(),
                text_utf8_bytes: document.text.len(),
            });
        }
        // The tokenizer can consume document.text here. This chapter only loads
        // prepared records; no preparation operation is repeated inside Rust.
    }
    Ok(CorpusSummary {
        first_documents,
        loaded: documents.progress(),
    })
}

With Docker running and the Rust workspace image available, load the output from the preceding NeMo command. Run from the repository root:

COURSE_CORPUS=false ./course run cargo run --offline --locked \
  -p practical-ch02-corpus-preparation \
  < .build/corpus/practical-ch02-exact-fixture/prepared.jsonl

The Rust summary should report $3$ loaded documents and $39$ decoded text bytes. Those bytes are $11$ for the cat story, $11$ for the dog story and $17$ for the rain story. Metadata and JSON punctuation are not included in that sum. The wrapper runs the Rust program without network access; COURSE_CORPUS=false avoids acquiring the optional large corpus while preparing the Rust image.

With Docker running and the Rust workspace image available, run from the repository root. For the tokenizer’s training input, select the training export created by the NeMo command instead of the all-role inspection file:

COURSE_CORPUS=false ./course run cargo run --offline --locked \
-p practical-ch02-corpus-preparation \
  < .build/corpus/practical-ch02-exact-fixture/train.jsonl

This summary reports $1$ document and $11$ decoded text bytes. Choosing that file is the caller’s responsibility: the loader does not infer a role from a filename or reject held-out records automatically.

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // stdin is supplied by the caller: a file redirect, pipe, or other source.
    // No HTTP, asset path, dataset choice, or GPU is built into this executable.
    let report = summarize(std::io::stdin().lock(), DEMO_LIMITS)?;
    println!("{}", serde_json::to_string_pretty(&report)?);
    Ok(())
}

History: more text was not the whole solution

Collecting readable text alone does not document its intended use, control the weight of repeated occurrences or protect held-out evaluation from training overlap.

Dataset documentation makes source context reviewable; deduplication studies connect repeated training text with memorization and train-test overlap in their measured settings.

Datasheets for Datasets was first posted in $2018$. Gebru and colleagues propose documenting dataset motivation, composition, collection and intended uses so those choices can be reviewed. Documentation records context; it does not itself establish permission or privacy.

The $2021$ C4 analysis examined where web-corpus text came from and what filtering removed. Its finding that blocklist filtering disproportionately excluded text from and about minority individuals is a reason to inspect exclusions, not to label every filter neutral.

In $2022$, Deduplicating Training Data Makes Language Models Better studied duplicate handling in training corpora. Lee and colleagues report reduced memorized output and train-test overlap after deduplication in their studied corpora and models; those outcomes are not measurements of this course fixture. They explain why duplicate handling and evaluation separation matter in addition to collecting and documenting text.

The following Rust comparison loads two identical supplied records and keeps both. It illustrates the limitation of merely reading raw occurrences, not a reimplementation of those studies’ preparation tools:

    #[test]
    fn loader_does_not_curate_repeated_ids_or_text() {
        let source =
            b"{\"id\":\"same\",\"text\":\"same text\"}\n{\"id\":\"same\",\"text\":\"same text\"}\n";
        let documents = PreparedCorpusReader::new(Cursor::new(source), limits())
            .unwrap()
            .collect::<Result<Vec<_>, _>>()
            .unwrap();
        assert_eq!(documents.len(), 2);
        assert_eq!(documents[0], documents[1]);
    }

Keep documented source and selection evidence, inspect exclusions, handle related records together and preserve held-out roles before tokenizer learning or model training. No single preparation statistic certifies suitability.

Follow the preparation-to-learning boundary

The preparation tool owns the configured exclusions and duplicate handling. The exported documents and their source and split evidence cross the boundary into the Rust training pipeline. The Rust loader reads that selection; the tokenizer learns only from the training role. Passing a record between stages does not grant it a new quality or privacy guarantee.

Optional practice: reproduce and explain the boundary

1. Load the manually prepared fixture with the first command below. Explain why matching its counts to the NeMo output does not identify its producer or establish that its text is suitable for training.
2. Run the malformed-later-input test with the second command. Explain what happens to the success summary and to the document returned before the error.

Run these commands from the repository root with Docker running. The supplied JSONL file is a separate, manually prepared course fixture; it is not a substitute for the preceding NeMo execution.

COURSE_CORPUS=false ./course run cargo run --offline --locked \
  -p practical-ch02-corpus-preparation \
  < rust/demos/practical-ch02-corpus-preparation/fixtures/prepared.jsonl

This command also prints $3$ loaded documents and $39$ decoded text bytes. Explain why equal counts do not prove that NeMo produced this file or that its content is suitable for training.

From the repository root, with Docker running and the Rust workspace image available, reproduce the loader’s failure boundary:

COURSE_CORPUS=false ./course run cargo test --offline --locked \
  -p practical-ch02-corpus-preparation --lib \
  tests::malformed_later_input_has_no_success_summary -- --exact

The test should report running 1 test and 1 passed; 0 failed. It supplies one valid document followed by malformed JSON and confirms that the demo produces no success summary. Zero selected tests means this check did not run.

What reaches the language model next

Prepared training documents become the tokenizer’s learning input in Chapter 3. Held-out documents may be encoded with the learned tokenizer, but they must not change its learned vocabulary or merge choices. Later batching creates token targets from that same frozen training selection. NeMo has not implemented any of those Rust LLM algorithms; it has supplied documented text for them to learn from.