04 · Content revision 9

Applying and reversing a BPE tokenizer

Replay frozen byte-pair ranks, reserve document controls, and recover every content byte exactly.

Predict the first merge before running it

Take a space followed by Cyrillic а:

text          = " а"
UTF-8 bytes   = [20, d0, b0]
trainer IDs   = [32, 208, 176]

Chapter 3 learned these two relevant rules:

Rank	Trainer-space pair	New trainer ID	Stored bytes
0	`(32,208)`	`256`	`20 d0`
1	`(208,176)`	`257`	`d0 b0`

Which rule wins on this input? Rank is priority, not a frequency to recompute. Rank 0 runs first and consumes 32,208, leaving [256,176]. Rank 1 can no longer see adjacent 208,176. After reserving two controls, every trainer ID is shifted by two:

canonical content IDs = [258, 178]
wrapped document IDs  = [0, 258, 178, 1]
                         ^             ^
                        BOS           EOS

Now reverse the content. Token 258 stores bytes 20 d0; token 178 is raw byte b0. Concatenating them gives 20 d0 b0, exactly the input. Notice that token 258 is not valid standalone UTF-8: its final d0 is a lead byte whose continuation arrives from the next token. Tokens are byte pieces, not guaranteed characters.

There is a subtle trap. [34,259] is also a valid content sequence: 34 stores the space byte 20, and rank-1 token 259 stores d0 b0. It decodes to the same three bytes. But re-encoding those bytes returns canonical [258,178], because rank 0 has priority. Exact decoding does not make every valid token sequence a canonical encoding.

Key idea: the tokenizer promises exact bytes for its own outputs; canonical encoding is a separate promise from accepting every decodable segmentation.

Guarantee exact bytes in one direction

The tokenizer promises:

\operatorname{decode}_{content}(\operatorname{encode}_{content}(x))=\operatorname{bytes}(x)

encode_content starts from one shifted ID per byte and replays the frozen rules from rank 0 upward. It does not inspect validation/test frequencies, learn a new pair, normalize Unicode, lowercase text, or split on words. decode_content looks up each content token’s stored byte expansion and concatenates the results.

The equation intentionally does not claim $\operatorname{encode}_{content}(\operatorname{decode}_{content}(z))=z$ for every token sequence $z$ . The worked pair [34,259] is a counterexample. This one-way statement is exactly what the model needs: one deterministic encoder and a lossless decoder for its outputs.

Symbol glossary and ID layout

Symbol	Meaning
$x$	the input content, supplied as UTF-8 text or directly as bytes
$\operatorname{encode}_{content}$	byte initialization followed by every frozen merge rank in ascending order, without BOS or EOS
$\operatorname{decode}_{content}$	concatenation of the stored byte expansion for every content token ID
$\operatorname{bytes}(x)$	the exact input byte sequence, with no Unicode normalization or replacement

Layout version 1 has one contiguous namespace:

IDs	Meaning	Mapping
`0`	BOS control	fixed
`1`	EOS control	fixed
`2..=257`	256 one-byte content tokens	byte $b$ maps to $b+2$
`258..`	learned content tokens	rank $r$ maps to $258+r$

There is no PAD ID in this course. The later training path uses fixed-length windows; padding-heavy serving is deliberately deferred. That is a scope choice, not a claim that production systems never pad.

Replace the unknown-string hole with byte coverage

A closed whole-word vocabulary can assign one token ID to each common fitted word, but an unseen spelling has no row. The historical Rust contrast maps both lowering and any other absent word to ID 0; decoding can recover only the marker <UNK>, not the original spelling:

A fixed whole-word vocabulary that cannot recover an unseen spelling rust/demos/ch04-apply-bpe-tokenizer/src/lib.rs#unknown-token-loss

pub fn fit_closed_word_vocabulary(documents: &[&str]) -> BTreeMap<String, u32> {
    documents
        .iter()
        .flat_map(|document| document.split_whitespace())
        .collect::<BTreeSet<_>>()
        .into_iter()
        .enumerate()
        .map(|(index, word)| (word.to_owned(), index as u32 + 1))
        .collect()
}

/// Encodes a spelling, collapsing every unseen word to the same ID.
pub fn encode_closed_word(vocabulary: &BTreeMap<String, u32>, word: &str) -> u32 {
    vocabulary.get(word).copied().unwrap_or(UNKNOWN_WORD_ID)
}

/// Decodes a known ID or the literal unknown marker; original unseen bytes are gone.
pub fn decode_closed_word(vocabulary: &BTreeMap<String, u32>, token_id: u32) -> String {
    vocabulary
        .iter()
        .find_map(|(word, &id)| (id == token_id).then(|| word.clone()))
        .unwrap_or_else(|| "<UNK>".to_owned())
}

Sennrich, Haddow, and Birch’s 2016 paper on rare words adapted repeated pair merging to character-sequence subwords. That creates useful units between whole words and characters, but that paper is not our evidence for a universal byte base.

The GPT-2 report’s input-representation section explains the byte-level step: UTF-8 bytes need a base alphabet of only 256 symbols and can represent any Unicode string. Coverage has a cost. A Cyrillic letter uses two UTF-8 bytes before merges; an emoji often uses four. A learned vocabulary may shorten frequent sequences, but byte-level coverage does not guarantee the smallest final vocabulary or equally short sequences for every script.

GPT-2 also applied category-boundary rules with a space exception. This course is not a clone of that tokenizer: it uses the exact Chapter 3 rules, permits any pair inside a document, and treats only document boundaries as barriers. Naming these differences prevents “BPE” from hiding important policy choices.

Historical takeaway: byte coverage solves the unknown-string problem, while rank policy and boundary policy determine the actual tokenizer behavior.

Build a frozen, strict tokenizer in Rust

The layout code makes the control/content separation executable rather than a comment. It validates the final u32 extent and exposes the byte and merge ranges:

Tokenizer layout version 1 with validated ID ranges rust/crates/llm-from-scratch/src/tokenizer/bpe.rs#token-id-layout

/// Serialized layout version taught by Chapter 4.
pub const TOKENIZER_LAYOUT_VERSION: u32 = 1;
/// Marks the beginning of one encoded document.
pub const BOS_TOKEN_ID: u32 = 0;
/// Marks the end of one encoded document.
pub const EOS_TOKEN_ID: u32 = 1;
/// Maps every Chapter 3 training-space ID into the content namespace.
pub const CONTENT_ID_OFFSET: u32 = 2;
/// First content ID representing one raw byte.
pub const FIRST_BYTE_TOKEN_ID: u32 = CONTENT_ID_OFFSET;
/// Last content ID representing one raw byte.
pub const LAST_BYTE_TOKEN_ID: u32 = CONTENT_ID_OFFSET + BYTE_TOKEN_COUNT - 1;
/// Content ID assigned to merge rank zero.
pub const FIRST_MERGE_TOKEN_ID: u32 = LAST_BYTE_TOKEN_ID + 1;

/// The fixed fields and vocabulary extent of tokenizer layout version 1.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct TokenizerLayout {
    merge_count: usize,
    vocabulary_size: usize,
}

impl TokenizerLayout {
    /// Validates that every content symbol can be represented by a `u32` ID.
    pub fn new(merge_count: usize) -> Result<Self, BpeTokenizerError> {
        let vocabulary_size = usize::try_from(FIRST_MERGE_TOKEN_ID)
            .ok()
            .and_then(|base| base.checked_add(merge_count))
            .ok_or(BpeTokenizerError::LayoutOverflow { merge_count })?;
        let highest_token = vocabulary_size
            .checked_sub(1)
            .and_then(|value| u32::try_from(value).ok())
            .ok_or(BpeTokenizerError::LayoutOverflow { merge_count })?;
        if merge_count > 0 && highest_token < FIRST_MERGE_TOKEN_ID {
            return Err(BpeTokenizerError::LayoutOverflow { merge_count });
        }
        Ok(Self {
            merge_count,
            vocabulary_size,
        })
    }

    /// Returns the serialized layout version.
    pub const fn version(self) -> u32 {
        TOKENIZER_LAYOUT_VERSION
    }

    /// Returns the number of frozen merge ranks.
    pub const fn merge_count(self) -> usize {
        self.merge_count
    }

    /// Returns the complete number of control and content IDs.
    pub const fn vocabulary_size(self) -> usize {
        self.vocabulary_size
    }

    /// Maps a raw byte to its one-byte content token.
    pub const fn byte_token_id(self, byte: u8) -> u32 {
        FIRST_BYTE_TOKEN_ID + byte as u32
    }

    /// Returns the final content ID assigned to a valid zero-based rank.
    pub fn merge_token_id(self, rank: usize) -> Option<u32> {
        if rank >= self.merge_count {
            return None;
        }
        usize::try_from(FIRST_MERGE_TOKEN_ID)
            .ok()
            .and_then(|base| base.checked_add(rank))
            .and_then(|token| u32::try_from(token).ok())
    }
}

BpeTraining is not an arbitrary pair table. BpeTrainer::train creates each rank, its assigned training-space token ID, and its byte expansion together. Its fields are private and its public accessors expose immutable views, so public callers cannot replace or mutate those values. BpeTokenizer::from_training copies each rule’s rank, pair, and assigned training-space ID plus the already- built vocabulary, while mapping each rule’s operands and result into layout version 1 by adding two. It does not reconstruct byte expansions or repeat the Chapter 3 invariants. It performs one check newly required by Chapter 4: after reserving BOS and EOS, the highest content ID implied by the number of learned rules must fit in u32. Because the tokenizer owns these copies, it remains usable after the training result is dropped.

BpeTokenizer::from_merge_pairs, by contrast, accepts raw pairs supplied by a caller. It first verifies that the complete layout fits in u32. At rank $r$ , it assigns training-space ID $256+r$ , requires both operands to have been defined before that rank, rejects a pair seen at an earlier rank, derives the new byte expansion by concatenating the stored operand expansions, and maps the operands and result into the content namespace by adding two. Exercises and later checkpoint loading use this fully checked path when they do not start from a BpeTraining value.

Encoding takes &[u8] as the fundamental input. The &str helper simply exposes its UTF-8 bytes. By the time encoding begins, both operands and the result of every rule have already been shifted by $2$ into the content-ID namespace. The encoder applies these stored rules once each in ascending rank order, using Chapter 3’s left-to-right replacement.

The ordinary and traced methods call the same ranked-merge loop. The ordinary method does not store before-and-after snapshots; it returns only the final token IDs. The traced method additionally copies the sequence before and after each rank that changed this input. Requesting those snapshots changes what the caller can inspect, not the encoded result:

Initialize byte IDs and replay every frozen rank rust/crates/llm-from-scratch/src/tokenizer/bpe.rs#ranked-content-encoding

    fn initial_content_tokens(&self, bytes: &[u8]) -> Vec<u32> {
        bytes
            .iter()
            .map(|byte| self.layout.byte_token_id(*byte))
            .collect()
    }

    fn apply_ranked_merges(
        &self,
        mut content_tokens: Vec<u32>,
        mut observe: impl FnMut(&BpeMergeRule, usize, &[u32], &[u32]),
    ) -> Vec<u32> {
        for rule in &self.merge_rules {
            let before = content_tokens;
            let (after, replacements) =
                replace_pair_left_to_right(&before, rule.content_pair, rule.content_token_id);
            if replacements > 0 {
                observe(rule, replacements, &before, &after);
            }
            content_tokens = after;
        }
        content_tokens
    }

    /// Encodes bytes and records every rank that changed the sequence.
    pub fn encode_content_with_trace(&self, bytes: &[u8]) -> BpeEncodingTrace {
        let initial_tokens = self.initial_content_tokens(bytes);
        let mut applications = Vec::new();
        let content_tokens = self.apply_ranked_merges(
            initial_tokens.clone(),
            |rule, replacements, before, after| {
                applications.push(BpeMergeApplication {
                    rank: rule.rank,
                    replacements,
                    before: before.to_vec(),
                    after: after.to_vec(),
                });
            },
        );

        BpeEncodingTrace {
            initial_tokens,
            applications,
            content_tokens,
        }
    }

    /// Encodes arbitrary bytes into the canonical rank-ordered content sequence.
    pub fn encode_content(&self, bytes: &[u8]) -> Vec<u32> {
        let initial_tokens = self.initial_content_tokens(bytes);
        self.apply_ranked_merges(initial_tokens, |_, _, _, _| {})
    }

    /// Encodes a valid UTF-8 string through the same byte boundary.
    pub fn encode_utf8(&self, text: &str) -> Vec<u32> {
        self.encode_content(text.as_bytes())
    }

Decoding fundamentally returns Vec<u8>. That makes all 256 fallback symbols reversible, including ff fe, which is not valid UTF-8. A separate helper asks for String and reports the exact invalid byte position instead of replacing it:

Recover bytes first; validate text only on request rust/crates/llm-from-scratch/src/tokenizer/bpe.rs#byte-exact-decoding

    /// Concatenates content-token expansions without interpreting them as text.
    pub fn decode_content(&self, content: &[u32]) -> Result<Vec<u8>, BpeTokenizerError> {
        for (position, &token_id) in content.iter().enumerate() {
            if token_id == BOS_TOKEN_ID || token_id == EOS_TOKEN_ID {
                return Err(BpeTokenizerError::ControlTokenInContent { position, token_id });
            }
        }
        self.decode_tokens(content, 0)
    }

    /// Decodes content bytes, then requires the result to be valid UTF-8.
    pub fn decode_content_utf8(&self, content: &[u32]) -> Result<String, BpeTokenizerError> {
        strict_utf8(self.decode_content(content)?)
    }

    /// Decodes a wrapped document, then requires the result to be valid UTF-8.
    pub fn decode_document_utf8(&self, document: &[u32]) -> Result<String, BpeTokenizerError> {
        strict_utf8(self.decode_document(document)?)
    }

Document controls are a structural layer. The encoder merges content first and only then adds [BOS, ..., EOS], so no rule can consume a control. The decoder requires both endpoints and rejects BOS/EOS inside:

Add endpoint controls after encoding and validate them strictly rust/crates/llm-from-scratch/src/tokenizer/bpe.rs#document-wrapping

    /// Encodes content first, then adds controls that never enter a merge pass.
    pub fn encode_document(&self, bytes: &[u8]) -> Vec<u32> {
        let content = self.encode_content(bytes);
        let mut document = Vec::with_capacity(content.len() + 2);
        document.push(BOS_TOKEN_ID);
        document.extend(content);
        document.push(EOS_TOKEN_ID);
        document
    }

    /// Encodes a UTF-8 document and adds its endpoint controls.
    pub fn encode_utf8_document(&self, text: &str) -> Vec<u32> {
        self.encode_document(text.as_bytes())
    }

    /// Validates one wrapped document and recovers its exact content bytes.
    pub fn decode_document(&self, document: &[u32]) -> Result<Vec<u8>, BpeTokenizerError> {
        if document.len() < 2 {
            return Err(BpeTokenizerError::DocumentTooShort {
                length: document.len(),
            });
        }
        if document[0] != BOS_TOKEN_ID {
            return Err(BpeTokenizerError::ExpectedBos { found: document[0] });
        }
        let last = document.len() - 1;
        if document[last] != EOS_TOKEN_ID {
            return Err(BpeTokenizerError::ExpectedEos {
                found: document[last],
            });
        }
        for (position, &token_id) in document[1..last].iter().enumerate() {
            if token_id == BOS_TOKEN_ID || token_id == EOS_TOKEN_ID {
                return Err(BpeTokenizerError::InteriorControlToken {
                    position: position + 1,
                    token_id,
                });
            }
        }
        self.decode_tokens(&document[1..last], 1)
    }

The runnable chapter retrains the exact eight Chapter 3 ranks from the training partition, freezes them, demonstrates the lossy historical table, exercises unseen and malformed bytes, exposes canonicalization, and prints the tokenizer layout, edge-case results, and two byte-exact round-trip examples:

Print the tokenizer layout and edge-case results rust/demos/ch04-apply-bpe-tokenizer/src/main.rs#chapter-output

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let corpus = Corpus::from_json(CORPUS_JSON)?;
    let manifest = SplitManifest::from_json(SPLIT_MANIFEST)?;
    let partitions = manifest.partition(&corpus)?;
    let training = BpeTrainer::new(8).train(&partitions)?;
    let tokenizer = BpeTokenizer::from_training(&training)?;
    let layout = tokenizer.layout();

    println!("layout version: {}", layout.version());
    println!("control ids: BOS={BOS_TOKEN_ID} EOS={EOS_TOKEN_ID}");
    println!(
        "content ranges: bytes={FIRST_BYTE_TOKEN_ID}..{LAST_BYTE_TOKEN_ID} merges={FIRST_MERGE_TOKEN_ID}..{} vocabulary={}",
        layout.vocabulary_size() - 1,
        layout.vocabulary_size()
    );
    println!("frozen merge ranks: {} (train only)", layout.merge_count());

    let historical = fit_closed_word_vocabulary(&["low lower", "new newest"]);
    let historical_id = encode_closed_word(&historical, "lowering");
    println!("historical input: lowering");
    println!("historical ids: {}", format_ids(&[historical_id]));
    println!(
        "historical decoded: {} (original bytes lost)",
        decode_closed_word(&historical, historical_id)
    );

    let unseen = "🦀";
    let unseen_ids = tokenizer.encode_utf8(unseen);
    println!("unseen UTF-8 input: {unseen}");
    println!("unseen content ids: {}", format_ids(&unseen_ids));
    println!(
        "unseen decoded: {}",
        tokenizer.decode_content_utf8(&unseen_ids)?
    );
    println!(
        "empty document ids: {}",
        format_ids(&tokenizer.encode_document(b""))
    );

    let malformed = [0xff, 0xfe];
    let malformed_ids = tokenizer.encode_content(&malformed);
    println!("malformed input bytes: {}", format_hex_array(&malformed));
    println!("malformed content ids: {}", format_ids(&malformed_ids));
    println!(
        "malformed decoded bytes: {}",
        format_hex_array(&tokenizer.decode_content(&malformed_ids)?)
    );
    println!(
        "malformed UTF-8: {}",
        tokenizer
            .decode_content_utf8(&malformed_ids)
            .expect_err("malformed bytes are not text")
    );
    println!(
        "interior control rejected: {}",
        tokenizer
            .decode_document(&[BOS_TOKEN_ID, 100, BOS_TOKEN_ID, EOS_TOKEN_ID])
            .expect_err("interior BOS must fail")
    );

    let canonical = tokenizer.encode_utf8(" а");
    let noncanonical = [34, 259];
    let same_bytes = tokenizer.decode_content(&noncanonical)?;
    println!("canonical \" а\" content ids: {}", format_ids(&canonical));
    println!("noncanonical same-byte ids: {}", format_ids(&noncanonical));
    println!(
        "noncanonical re-encodes as: {}",
        format_ids(&tokenizer.encode_content(&same_bytes))
    );

    println!("TRACE apply-bpe-tokenizer-v1 BEGIN");
    println!(
        "LAYOUT version={} bos={} eos={} content_offset=2 byte_count=256 merge_count={} vocabulary_size={}",
        layout.version(),
        BOS_TOKEN_ID,
        EOS_TOKEN_ID,
        layout.merge_count(),
        layout.vocabulary_size()
    );
    for rule in tokenizer.merge_rules() {
        println!(
            "RULE rank={} training_pair={},{} training_token={} content_pair={},{} content_token={} bytes_hex={}",
            rule.rank(),
            rule.training_pair().left(),
            rule.training_pair().right(),
            rule.training_token_id(),
            rule.content_pair().left(),
            rule.content_pair().right(),
            rule.content_token_id(),
            format_hex(
                tokenizer
                    .token_bytes(rule.content_token_id())
                    .expect("merge token has bytes")
            )
        );
    }
    print_trace_case(&tokenizer, "ascii-bee", b"bee ")?;
    print_trace_case(&tokenizer, "cyrillic-a", " а".as_bytes())?;
    println!("TRACE apply-bpe-tokenizer-v1 END");
    println!("chapter 5 handoff: preserve each wrapped document boundary");
    Ok(())
}

BpeTokenizer also handles empty content, every possible byte value, ASCII, Cyrillic, emoji, NUL/newline mixtures, and malformed UTF-8. The BpeTokenizerError variants distinguish an overflowing ID layout, a merge that refers to an operand not yet defined, a duplicate pair, an unknown token ID, and each invalid control-token position. These boundary cases matter because exact byte recovery includes data that is not valid text, while document structure must still reject misplaced controls.

Follow grouping and exact inverse concatenation

Ranked byte groups reverse to the exact input

Follow one ASCII and one Cyrillic input through canonical content IDs and document controls, then follow the pipeline back and concatenate stored bytes in token order.

Layout version: 1
BOS: 0
EOS: 1
Content offset: 2

ASCII example: bee plus a space

Input text "bee "
1. UTF-8 bytes (hex)
  
  62656520
2. ↓
3. Shifted byte-token content IDs
  
  10010310334
4. ↓
5. Canonical ranked groups
  
  Applied rank: 7
  1. Token ID
    
    100
    
    Stored bytes
    
    62
    
    One-byte fallback
  2. Token ID
    
    103
    
    Stored bytes
    
    65
    
    One-byte fallback
  3. Token ID
    
    265
    
    Stored bytes
    
    65 20
    
    Applied rank 7
6. ↓
7. Document IDs
  
  BOS:0 100 103 265 EOS:1
  
  BOS — beginning boundary EOS — ending boundary
8. ↓
9. Recovered bytes (hex)
  
  62656520
  
  Exact byte match
Cyrillic example: a space plus а

Input text " а"
1. UTF-8 bytes (hex)
  
  20d0b0
2. ↓
3. Shifted byte-token content IDs
  
  34210178
4. ↓
5. Canonical ranked groups
  
  Applied rank: 0
  1. Token ID
    
    258
    
    Stored bytes
    
    20 d0
    
    Applied rank 0
  2. Token ID
    
    178
    
    Stored bytes
    
    b0
    
    One-byte fallback
6. ↓
7. Document IDs
  
  BOS:0 258 178 EOS:1
  
  BOS — beginning boundary EOS — ending boundary
8. ↓
9. Recovered bytes (hex)
  
  20d0b0
  
  Exact byte match

What both pipelines prove

Frozen ranks run in ascending order; the input never changes their priority.
$+2$ Every content ID is its Chapter 3 training ID plus two.
BOS and EOS appear only after encoding and only at document endpoints.
Stored piece bytes concatenate to the exact input without normalization.

Both examples show results computed by the Rust program. In bee , rank 7 groups the last e and space as token 265; the preceding b and e remain one-byte fallbacks. In а, rank 0 groups a space with the first byte of а, leaving its continuation byte separate. That second case makes it impossible to pretend that token boundaries are character boundaries.

Read each pipeline downward to encode, then inspect each piece’s stored bytes to decode. BOS and EOS are outside the content pieces. The check mark, border, text, and byte values all carry the equality cue, so color is never the only signal. In both examples, concatenating the decoded content bytes recovers the original byte sequence exactly.

Predict, then check

For bytes 20 d0 b0, predict the result if rank 1 were incorrectly applied before rank 0. Then give the canonical result.
Map ASCII ? (byte 3f), trainer merge ID 257, and merge rank 7 into layout version 1.
Predict encode_content("") and encode_document("").
Predict the content IDs for unseen 🦀 with no matching learned pair.
Cyrillic т is UTF-8 d1 82, and rank 2 learned (209,130). Predict its final content sequence.
Decode [34,259], then predict the result of re-encoding those bytes.
Identify the first error in document [0,100,0,1] and content [100,1].
Decide whether exact byte decoding of [257,256] must produce a valid String.

Check your predictions

Wrong order would create trainer [32,257], shifted [34,259]. Correct rank order creates [256,176], shifted [258,178].
3f is decimal 63, so ? maps to 65. Trainer ID 257 maps to 259; rank 7 maps to $258+7=265$ .
Empty content is []; its document is [0,1]. Both decode to zero bytes.
🦀 is f0 9f a6 80, so the shifted fallback IDs are [242,161,168,130].
Shifted operands (211,132) merge at rank 2 into ID 260, so the result is [260].
[34,259] expands to 20 d0 b0, or " а". Re-encoding applies rank 0 first and returns [258,178], not the original token sequence.
The BOS at document position 2 is an interior control. EOS at content position 1 is forbidden because content decoding accepts no control IDs.
No. IDs [257,256] recover bytes ff fe exactly; strict UTF-8 conversion rejects them at byte 0. Exact byte recovery is the primary guarantee.

Misconception check: “If decoding is lossless, every valid token sequence is the encoder output for its bytes.” False. Lossless decoding permits multiple segmentations; frozen rank priority chooses one canonical encoding.

Preserve one boundary-aware sequence per document

The cumulative implementation can now transform every source document into one deterministic sequence [BOS, content..., EOS] and recover the original bytes. Merge statistics still come only from Chapter 2’s training documents; applying the frozen tokenizer to validation or test content does not refit it.

Chapter 5 receives these wrapped sequences separately. It will create shifted context/target windows inside each one, never concatenate two documents, and never cross a train/validation/test boundary. BOS and EOS therefore become part of the causal sequence without becoming mergeable text: BOS anchors the initial context, while EOS is the terminal prediction target.

Carry forward: Chapter 5 can now train on complete, reversible, boundary-aware token sequences without changing their content bytes.