Text units and vocabulary IDs

Give text units fixed labels

Before text can enter the model described in Chapter 0, it must become a sequence of integer IDs drawn from a finite vocabulary. A tokenizer chooses the text units; a vocabulary associates those units with IDs. The choice of unit determines both what the table must contain and how long the ID sequence becomes. This lesson uses one Unicode scalar value per token so that every lookup is small enough to inspect.

Three different units matter before we count anything:

• A byte is an eight-bit storage unit. Rust strings store text as UTF-8 bytes; a single Unicode scalar value may occupy several of those bytes.
• A Unicode scalar value is a Unicode code point other than a surrogate code point. Unicode uses code points as numeric values; Rust’s char represents one scalar value. See the Rust char documentation.
• A grapheme cluster approximates a character as a reader perceives it. It can contain several scalars: é, written as U+0065 followed by the combining acute accent U+0301, forms one grapheme cluster from two scalars. These boundaries are described in Unicode’s text-segmentation specification.

Rust’s str::chars iterates scalar values, so this demo does not count or segment grapheme clusters. The notation U+... writes a Unicode value in hexadecimal. The byte arrays and token-ID arrays below use decimal numbers.

Build the table once

The demo’s only training text is cat кот, including the space. Construction collects its distinct scalars, sorts them by numeric Unicode value, and assigns known IDs starting at 1. ID 0 is reserved for the unknown marker <UNK>.

Sorting makes these assignments reproducible; it is this demo’s construction rule, not a required ordering for all tokenizers. The table stays fixed during encoding: an input lookup neither learns a new unit nor adds an entry. Rebuilding a vocabulary from different training text can change the assignments.

An ID is a table label. For example, c has Unicode notation U+0063, UTF-8 byte [99], and vocabulary ID 3. These numbers describe different representations. IDs that happen to be numerically close do not thereby represent similar meanings, and the raw IDs are not learned embeddings.

Read the inputs through the fixed table

For cat, look up c, then a, then t: the resulting IDs are [3, 2, 4]. For кот, look up к, then о, then т: the IDs are [5, 6, 7]. Encoding preserves input order even though the vocabulary itself was sorted.

In cat, each scalar contributes one byte. In кот, к contributes [208, 186], о contributes [208, 190], and т contributes [209, 130]. Each byte group represents one scalar, which receives one ID. Thus cat has 3 bytes, 3 scalars, and 3 IDs; кот has 6 bytes, 3 scalars, and 3 IDs.

Key idea: Storage size and token-sequence length count different units. This demo emits one ID per scalar; another tokenizer may choose different units.

One lookup at each scalar position

The worked examples apply the same rule at every position:

At position $i$, the input scalar $u_i$ goes through lookup $V$, and its ID $z_i$ is placed at the same position in the output. The set $S$ contains exactly the seven known scalars from cat кот, including the space. If a scalar is absent from $S$, lookup returns the reserved ID $0$.

Positions are counted from zero. In cat, $i=0$ identifies the first position, whose scalar is c and whose ID is 3. The position tells us where a value occurs; the ID tells us which vocabulary entry it names. One lookup per scalar keeps the scalar and ID sequences equally long, regardless of the number of UTF-8 bytes.

Each known scalar has its own ID, so decoding known IDs in order recovers the original scalar sequence. That is why both cat and кот survive encoding followed by decoding. The same reasoning applies to any sequence made entirely from this table’s known scalars.

The unknown input ? marks the limit of that guarantee. Its UTF-8 bytes are [63] and its scalar is U+003F, but that scalar is absent from the table. Encoding therefore produces [0]; decoding produces the literal text <UNK>. All unknown scalars share that one ID, so the decoder cannot recover which scalar was replaced. The loss comes from vocabulary coverage, even though ? is valid UTF-8. Supplying an ID outside the table, such as 8, is a separate error that the existing decoder rejects.

Symbols in the lookup rule

Why use units smaller than words

A whole-word vocabulary needs separate entries for distinct forms. If a new form is absent, a fixed whole-word table must treat it as unknown. Splitting into smaller known units can compose a form the table has never stored as a whole, but a multi-letter word then takes more sequence positions.

The existing Rust contrast starts with whitespace boundaries. red fox becomes ["red", "fox"], and рыжий кот becomes ["рыжий", "кот"]. Rust’s split_whitespace splits at whitespace; punctuation remains attached. This is a simple illustration of word-sized units rather than a complete linguistic word segmenter.

/// Demonstrates the historical intuition of whitespace-delimited word units.
///
/// This is a contrast for the lesson, not a general-purpose tokenizer:
/// punctuation stays attached and unseen word forms remain distinct.
pub fn split_words(text: &str) -> Vec<&str> {
    text.split_whitespace().collect()
}

/// Demonstrates scalar-level units, historically called character-level units.
pub fn split_scalars(text: &str) -> Vec<char> {
    text.chars().collect()
}

By contrast, split_scalars("кот") returns ['к', 'о', 'т']: three units for one word. Smaller units reduce the need for whole-form entries, but a scalar vocabulary still loses unseen scalars and still does not identify grapheme clusters. Vocabulary size, text coverage, and sequence length are separate parts of the tradeoff; scalar splitting does not settle all three.

Sennrich, Haddow, and Birch (2016) adapted byte-pair encoding, or BPE, for subword segmentation in neural machine translation. Their method starts from character sequences and learns units that can combine into words. That work motivates a way to balance vocabulary size and sequence length; it is not the byte-based tokenizer built later in this course.

Read the existing Rust example

The small demo exposes the operations just explained. as_bytes() returns the UTF-8 storage bytes; chars() yields the decoded Unicode scalar values in order:

/// Copies the UTF-8 code units that make up `text`.
pub fn utf8_bytes(text: &str) -> Vec<u8> {
    text.as_bytes().to_vec()
}

/// Decodes `text` into Unicode scalar values in source order.
pub fn unicode_scalars(text: &str) -> Vec<char> {
    text.chars().collect()
}

On the same text, text.len() counts UTF-8 bytes, while text.chars().count() counts Unicode scalar values. Neither operation counts grapheme clusters.

Vocabulary::from_training_text sorts and deduplicates the scalars. The stored order determines known IDs and lets the decoder look up their scalars again. encode performs one lookup for each scalar; decode recovers known scalars, writes <UNK> for 0, and rejects an out-of-range ID.

/// A fixed mapping from Unicode scalar values to deterministic integer IDs.
///
/// ID `0` is always [`UNKNOWN_TOKEN_ID`]. Known scalar values are sorted by
/// their numeric Unicode value and receive consecutive IDs beginning at `1`.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Vocabulary {
    known_units: Vec<char>,
}

impl Vocabulary {
    /// Builds a vocabulary from the unique scalar values in `training_text`.
    pub fn from_training_text(training_text: &str) -> Self {
        let mut known_units = unicode_scalars(training_text);
        known_units.sort_unstable();
        known_units.dedup();
        Self { known_units }
    }

    /// Returns known units in their deterministic ID order.
    pub fn known_units(&self) -> &[char] {
        &self.known_units
    }

    /// Iterates over `(ID, scalar)` pairs, excluding the reserved unknown ID.
    pub fn entries(&self) -> impl Iterator<Item = (usize, char)> + '_ {
        self.known_units
            .iter()
            .copied()
            .enumerate()
            .map(|(index, unit)| (index + 1, unit))
    }

    /// Looks up one scalar value, returning [`UNKNOWN_TOKEN_ID`] when absent.
    pub fn id_for(&self, unit: char) -> usize {
        self.known_units
            .binary_search(&unit)
            .map_or(UNKNOWN_TOKEN_ID, |index| index + 1)
    }

    /// Looks up one ID.
    ///
    /// `Ok(None)` represents the reserved unknown token. An ID above the
    /// vocabulary's largest known ID is an error rather than an unknown token.
    pub fn unit_for_id(&self, id: usize) -> Result<Option<char>, InvalidTokenId> {
        if id == UNKNOWN_TOKEN_ID {
            return Ok(None);
        }

        self.known_units
            .get(id - 1)
            .copied()
            .map(Some)
            .ok_or(InvalidTokenId {
                id,
                max_id: self.known_units.len(),
            })
    }

    /// Encodes one token ID per Unicode scalar value in `text`.
    pub fn encode(&self, text: &str) -> Vec<usize> {
        text.chars().map(|unit| self.id_for(unit)).collect()
    }

    /// Decodes IDs, rendering ID `0` as the literal [`UNKNOWN_TOKEN`].
    pub fn decode(&self, ids: &[usize]) -> Result<String, InvalidTokenId> {
        let mut text = String::with_capacity(ids.len());
        for &id in ids {
            match self.unit_for_id(id)? {
                Some(unit) => text.push(unit),
                None => text.push_str(UNKNOWN_TOKEN),
            }
        }
        Ok(text)
    }
}

This Vocabulary belongs to the ch01-text-units demo. The cumulative llm-from-scratch crate does not import or extend it. Its role here is to make the unit-to-ID mapping and the unknown-scalar loss observable.

The executable uses the fixed training text and inputs from the worked example:

fn main() -> Result<(), InvalidTokenId> {
    let vocabulary = Vocabulary::from_training_text(TRAINING_TEXT);

    print!("vocabulary: {UNKNOWN_TOKEN}={UNKNOWN_TOKEN_ID}");
    for (id, unit) in vocabulary.entries() {
        print!(" {unit:?}={id}");
    }
    println!();

    print_example(&vocabulary, ENGLISH_INPUT)?;
    print_example(&vocabulary, CYRILLIC_INPUT)?;

    println!(
        "historical words: {:?} | {:?}",
        split_words("red fox"),
        split_words("рыжий кот")
    );
    println!(
        "historical scalars: {:?} | {:?}",
        split_scalars(ENGLISH_INPUT),
        split_scalars(CYRILLIC_INPUT)
    );

    let unknown_ids = vocabulary.encode(UNKNOWN_INPUT);
    println!("unknown input: {UNKNOWN_INPUT}");
    println!("unknown token ids: {unknown_ids:?}");
    println!("unknown decoded: {}", vocabulary.decode(&unknown_ids)?);

    Ok(())
}

Optionally reproduce these results from the repository root, with Cargo and Rust 1.93 or newer available, as required by the repository:

cargo test --locked -p ch01-text-units
cargo run --quiet -p ch01-text-units | diff -u rust/demos/ch01-text-units/expected.txt -

The focused package tests report 6 passing library tests. They cover the exact representations, sorted IDs, known round trips, empty input, unknown and invalid IDs, and the two splitting functions. The second command compares the fixed program output with the committed result; success prints nothing and exits with status zero.

Follow each scalar from bytes to its ID

The diagram shows the already explained cat and кот examples. Each scalar stays aligned with its UTF-8 byte group and its vocabulary ID. Read the numbered stages in order; brackets and byte-count labels identify groups without relying on color.

Both examples have three scalar groups and three IDs. Their byte counts differ because each Cyrillic scalar here needs two bytes. This alignment belongs to the demo’s choice of one scalar per token; it is not a rule for every tokenizer.

Carry the mapping concept into the model

Later, each token ID selects one row from an embedding table: a stored vector whose values the model learns. The model uses those vectors to compute one score for each possible next ID; a decoding policy chooses the next ID from those scores. That output must be decoded with the matching vocabulary. Changing the tokenizer’s ID assignments independently of the model weights changes what the embedding rows mean, even when the integers themselves remain valid.

The next chapters replace this scalar demo in a deliberate order. Chapter 2 preserves document boundaries and freezes the corpus partitions. Chapter 3 builds a separate BPE vocabulary with a base token for each of all 256 byte values and adds learned merge tokens that can represent several bytes. Chapter 4 reserves BOS and EOS, control tokens marking the beginning and end of a document, shifts the content IDs into its own namespace, and applies the frozen tokenizer.

Because every UTF-8 byte has a base token, unfamiliar Unicode scalars can still be represented through their bytes. The new tokenizer does not inherit the demo’s rule that an unseen scalar becomes <UNK>. The scalar table, its ID assignments, and its concrete Vec<usize> stay in Chapter 1; the idea that a fixed vocabulary gives meaning to an ordered ID sequence carries forward.