Text units and vocabulary IDs

Start with two three-letter inputs

Rust stores both cat and кот as UTF-8, but equal-looking lengths do not imply equal byte lengths. Before reading the table, predict three counts for each input: its bytes, its Unicode scalar values, and its vocabulary IDs.

The result is 3 / 3 / 3 for cat and 6 / 3 / 3 for кот. Each Cyrillic scalar occupies two UTF-8 bytes here, but both inputs still contain three scalar positions and produce three IDs.

The IDs come from one fixed training string, cat кот. ID 0 is reserved for <UNK>; known scalar values are sorted by numeric Unicode value and receive IDs from 1:

This rule is deterministic: it does not depend on hash-map iteration order. The round trip is reversible for known units. For the unseen input ?, byte 63 and scalar U+003F are observable before lookup, but encoding produces [0] and decoding returns <UNK> rather than the original question mark.

The concrete Vocabulary type is a deliberately small Chapter 1 comparison, not the tokenizer used by later chapters. It assigns a distinct ID to each known Unicode scalar value. Its fixed table therefore makes the mapping easy to inspect, while its single <UNK> ID also exposes the cost of incomplete coverage.

Key idea: UTF-8 byte count and token-sequence length measure different units. This demo emits one ID per Unicode scalar, so its token-sequence length equals its scalar count, while only scalars present in the fixed vocabulary survive a round trip.

One lookup per scalar position

For every scalar position, encoding applies the same vocabulary lookup:

In plain language, take the scalar value at position $i$, look it up in the fixed vocabulary, and place the resulting integer ID at the same position. A scalar may use several UTF-8 bytes, but it contributes exactly one position to this chapter’s token sequence. The inverse lookup recovers every known scalar; ID $0$ deliberately recovers only the marker <UNK>.

Symbol glossary

Before subword tokenizers

A straightforward early approach was a vocabulary of whitespace-delimited whole words: red fox becomes ["red", "fox"]. It is easy to explain, but each unseen spelling, inflection, or punctuation-attached form needs a new entry or becomes unknown.

Character-level models moved to much smaller units. In this lesson, “character level” specifically means the Unicode scalar values returned by Rust’s str::chars; it does not mean user-perceived grapheme clusters. Scalar sequences avoid whole-word unknowns but are longer, and a visible grapheme can still contain more than one scalar.

/// Demonstrates the historical intuition of whitespace-delimited word units.
///
/// This is a contrast for the lesson, not a general-purpose tokenizer:
/// punctuation stays attached and unseen word forms remain distinct.
pub fn split_words(text: &str) -> Vec<&str> {
    text.split_whitespace().collect()
}

/// Demonstrates scalar-level units, historically called character-level units.
pub fn split_scalars(text: &str) -> Vec<char> {
    text.chars().collect()
}

Try the two functions mentally before running them: split_words("red fox") yields ["red", "fox"], while split_scalars("кот") yields ['к', 'о', 'т']. Modern subword methods seek a useful compromise between huge whole-word vocabularies and long scalar sequences. Chapter 2 first preserves documents and freezes the data partitions; chapter 3 learns the BPE merge rules, and chapter 4 applies the frozen tokenizer.

Carry forward: the next tokenizer must shorten sequences without giving up the ability to represent unfamiliar text.

Implement the mapping in Rust

The demo uses only Rust’s standard library. First, make the byte/scalar distinction explicit instead of hiding it behind the word “character”:

/// Copies the UTF-8 code units that make up `text`.
pub fn utf8_bytes(text: &str) -> Vec<u8> {
    text.as_bytes().to_vec()
}

/// Decodes `text` into Unicode scalar values in source order.
pub fn unicode_scalars(text: &str) -> Vec<char> {
    text.chars().collect()
}

Vocabulary::from_training_text then sorts and deduplicates the scalar values. Because the sorted vector is both the ID order and the inverse table, known units can travel in either direction without a tokenization library:

/// A fixed mapping from Unicode scalar values to deterministic integer IDs.
///
/// ID `0` is always [`UNKNOWN_TOKEN_ID`]. Known scalar values are sorted by
/// their numeric Unicode value and receive consecutive IDs beginning at `1`.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Vocabulary {
    known_units: Vec<char>,
}

impl Vocabulary {
    /// Builds a vocabulary from the unique scalar values in `training_text`.
    pub fn from_training_text(training_text: &str) -> Self {
        let mut known_units = unicode_scalars(training_text);
        known_units.sort_unstable();
        known_units.dedup();
        Self { known_units }
    }

    /// Returns known units in their deterministic ID order.
    pub fn known_units(&self) -> &[char] {
        &self.known_units
    }

    /// Iterates over `(ID, scalar)` pairs, excluding the reserved unknown ID.
    pub fn entries(&self) -> impl Iterator<Item = (usize, char)> + '_ {
        self.known_units
            .iter()
            .copied()
            .enumerate()
            .map(|(index, unit)| (index + 1, unit))
    }

    /// Looks up one scalar value, returning [`UNKNOWN_TOKEN_ID`] when absent.
    pub fn id_for(&self, unit: char) -> usize {
        self.known_units
            .binary_search(&unit)
            .map_or(UNKNOWN_TOKEN_ID, |index| index + 1)
    }

    /// Looks up one ID.
    ///
    /// `Ok(None)` represents the reserved unknown token. An ID above the
    /// vocabulary's largest known ID is an error rather than an unknown token.
    pub fn unit_for_id(&self, id: usize) -> Result<Option<char>, InvalidTokenId> {
        if id == UNKNOWN_TOKEN_ID {
            return Ok(None);
        }

        self.known_units
            .get(id - 1)
            .copied()
            .map(Some)
            .ok_or(InvalidTokenId {
                id,
                max_id: self.known_units.len(),
            })
    }

    /// Encodes one token ID per Unicode scalar value in `text`.
    pub fn encode(&self, text: &str) -> Vec<usize> {
        text.chars().map(|unit| self.id_for(unit)).collect()
    }

    /// Decodes IDs, rendering ID `0` as the literal [`UNKNOWN_TOKEN`].
    pub fn decode(&self, ids: &[usize]) -> Result<String, InvalidTokenId> {
        let mut text = String::with_capacity(ids.len());
        for &id in ids {
            match self.unit_for_id(id)? {
                Some(unit) => text.push(unit),
                None => text.push_str(UNKNOWN_TOKEN),
            }
        }
        Ok(text)
    }
}

This Vocabulary is defined only in the ch01-text-units demo; the cumulative llm-from-scratch crate neither imports nor extends it. Four ideas do carry forward: freeze one mapping between token IDs and represented units, make encoding deterministic, decode represented units exactly, and choose the coverage versus sequence-length tradeoff deliberately. The scalar entries, their numeric IDs, and the scalar-level <UNK> rule are not part of the later BPE tokenizer.

The executable fixes the inputs and prints every intermediate representation:

fn main() -> Result<(), InvalidTokenId> {
    let vocabulary = Vocabulary::from_training_text(TRAINING_TEXT);

    print!("vocabulary: {UNKNOWN_TOKEN}={UNKNOWN_TOKEN_ID}");
    for (id, unit) in vocabulary.entries() {
        print!(" {unit:?}={id}");
    }
    println!();

    print_example(&vocabulary, ENGLISH_INPUT)?;
    print_example(&vocabulary, CYRILLIC_INPUT)?;

    println!(
        "historical words: {:?} | {:?}",
        split_words("red fox"),
        split_words("рыжий кот")
    );
    println!(
        "historical scalars: {:?} | {:?}",
        split_scalars(ENGLISH_INPUT),
        split_scalars(CYRILLIC_INPUT)
    );

    let unknown_ids = vocabulary.encode(UNKNOWN_INPUT);
    println!("unknown input: {UNKNOWN_INPUT}");
    println!("unknown token ids: {unknown_ids:?}");
    println!("unknown decoded: {}", vocabulary.decode(&unknown_ids)?);

    Ok(())
}

Run the behavioral tests and compare stdout with the committed result:

cargo test --workspace --locked
cargo run --quiet -p ch01-text-units | diff -u rust/demos/ch01-text-units/expected.txt -

The tests cover deterministic ordering, exact English and Cyrillic values, known round trips, empty input, unknown and invalid IDs, and both historical split functions. A successful diff prints nothing.

Trace each position across representations

The diagram groups the bytes that belong to each scalar, then keeps that scalar aligned with its final ID. Read the numbered stages in order. The brackets and byte-count labels carry the grouping without relying on color.

Notice the structural invariant: both examples have three scalar groups and three IDs. Only the number of bytes inside each group changes. The unseen ? would still have a byte and scalar stage, but its final stage would contain the reserved ID 0.

Predict, then check

1. Without running the demo, write the byte/scalar/ID counts for cat and кот.
2. Use numeric scalar sorting to predict the IDs for both inputs.
3. Decide what information remains observable, and what is lost, when ? becomes ID 0.
4. Explain why text.len(), text.chars().count(), and the number of user-perceived graphemes can disagree.
5. Predict the historical word and scalar lines printed by the executable.

Change one input only after the fixed example passes. When you add a scalar that is absent from cat кот, predict 0 before confirming it with encode.

The boundary the decoder will consume

This chapter demonstrates a deterministic text-to-token-ID-sequence boundary; it does not contribute the concrete Rust ID type or token table used by the cumulative model. In that model, each ID selects one row from an embedding table before the sequence enters the decoder. The decoder eventually predicts another ID, and the inverse mapping for that same vocabulary turns generated IDs back into text.

Chapter 2 first preserves document boundaries and freezes the data partitions. Chapter 3 then creates a separate vocabulary for BPE training: it begins with one token for each of the 256 possible byte values and appends learned merge tokens that may represent several bytes. Chapter 4 reserves BOS and EOS, shifts those content IDs into a new token-ID namespace, and applies the frozen byte-level tokenizer. Every UTF-8 byte has a base token, so this tokenizer does not reuse Chapter 1’s rule that an unseen scalar becomes <UNK>.

The concrete scalar Vocabulary is therefore replaced, not extended. The requirement to freeze a mapping and the integer-sequence boundary remain: embeddings, attention, and next-token prediction can operate on IDs without needing to know which text unit each vocabulary chose.