05 · Content revision 8

Building autoregressive input–target pairs

Build next-token input–target pairs inside one encoded document at a time.

Predict all three input–target pairs before running the code

An autoregressive language model predicts each token from earlier tokens. This chapter builds the input–target pairs that provide the correct next-token answers without joining unrelated documents. The pairs define what to predict; the model will still need a separate rule that prevents it from looking ahead.

Chapter 4 produced one sequence with BOS and EOS for every document. Start with this tiny one:

position  0   1   2   3   4   5
token     0  41  42  43  44   1
          ^                   ^
         BOS                 EOS

Choose context length $T=3$ and stride $S=1$ . One pair needs four source tokens: three for the input and one more to shift the target. Before you read on, write the input and target at starts 0, 1, and 2.

At start 0, take source positions 0 through 3. The first three become the input; the last three become the target:

start 0   input  [0, 41, 42]
          target    [41, 42, 43]

Moving one position at a time gives the complete set:

Start	Input $x$	Target $y$
`0`	`[0,41,42]`	`[41,42,43]`
`1`	`[41,42,43]`	`[42,43,44]`
`2`	`[42,43,44]`	`[43,44,1]`

Each row is one autoregressive pair. Its input and target together form one autoregressive example; the Rust API represents it as a CausalWindow.

The final target includes EOS. That is intentional: after token 44, the model should learn that this document ends. At the next candidate start, 3, only [43,44,1] remains. A pair requires four source tokens, so there is no fourth pair; we do not invent padding tokens to complete it.

“No pair starts at 3” does not mean that tokens 43, 44, and EOS are thrown away. All three already occur in the complete pairs above. This suffix is simply too short to begin another pair.

Express the one-token shift with slices

For one encoded document $z$ , a valid candidate start $s$ produces:

x^{(s)}=z_{s:s+T}, \quad y^{(s)}=z_{s+1:s+T+1}

Rust ranges are half-open, so both slices contain exactly $T$ IDs. The target starts one position later: target element $i$ is the source token immediately after input element $i$ . A start is valid exactly when $s+T<|z|$ , or, equivalently, when $z$ contains all $T+1$ source tokens from that start.

The stride selects candidate starts $s=kS$ for $k=0,1,2,\ldots$ . With $S=2$ , the worked sequence considers starts 0, 2, and then 4. Starts 0 and 2 produce pairs; start 1 is skipped by the stride, while start 4 leaves only [44,1], too few tokens for another pair.

$T$ is the input length, the target length, and the maximum context available inside one pair. At target position $i$ , causal computation may use only $x_0$ through $x_i$ , not later input positions. $S$ is the distance between candidate starts: $S=1$ considers every possible start and repeats some adjacent transitions across overlapping pairs. A larger stride produces fewer pairs, skips some otherwise complete starts, and can leave gaps when $S>T$ . Restarting for every document avoids teaching an invented transition between unrelated texts. Keeping the partitions separate preserves the fixed split so later fitting code can choose training documents explicitly.

The shifted pair says which next token is correct at each position, but it does not stop a model from reading later entries of $x$ . A left-to-right recurrent model enforces that limit through sequential state. A decoder using self-attention needs an explicit causal mask.

What each symbol means

Symbol	Meaning in this chapter
$z$	exactly one encoded `[BOS, content..., EOS]` document
$T$	positive context length; the input and target each contain $T$ IDs
$S$	positive distance between candidate starts
$s$	one candidate start $kS$ ; it yields a pair only when $T+1$ source tokens remain
$x^{(s)}$	input slice beginning at $s$
$y^{(s)}$	target slice beginning at $s+1$

Apply the rule to each document separately; do not concatenate a partition into one global $z$ . A short document may produce no pair and leave only a suffix that is too short for one. Even a document with empty content is encoded as [BOS,EOS]: with $T=1$ , it produces the valid pair [BOS] -> [EOS].

Derive next-token targets from the sequence itself

In a classification task, each answer is supplied separately from its input. The Rust demonstration contains two token sequences with human-assigned sentiment labels. Those labels are not next tokens:

Two token sequences with separately supplied sentiment labels rust/demos/ch05-autoregressive-examples/src/lib.rs#hand-labeled-contrast

/// One task-specific row whose label had to be supplied separately.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct HandLabeledRow<'a> {
    /// Fixed input chosen by the example author.
    pub input: &'a [u32],
    /// Separate sentiment class attached to that input.
    pub label: SentimentLabel,
}

/// A human-supplied class for the synthetic sentiment contrast.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum SentimentLabel {
    /// The example expresses negative sentiment.
    Negative,
    /// The example expresses positive sentiment.
    Positive,
}

impl SentimentLabel {
    /// Returns the label as display text.
    pub const fn as_str(self) -> &'static str {
        match self {
            Self::Negative => "negative",
            Self::Positive => "positive",
        }
    }
}

/// Returns two tiny task-specific rows with separately supplied labels.
pub const fn hand_labeled_rows() -> [HandLabeledRow<'static>; 2] {
    [
        HandLabeledRow {
            input: &[41, 42, 43],
            label: SentimentLabel::Negative,
        },
        HandLabeledRow {
            input: &[51, 52, 53],
            label: SentimentLabel::Positive,
        },
    ]
}

Language modeling has a different source of supervision: the observed next token already follows every context that has a successor. Bengio et al. (2003) factor a sentence probability into next-word conditional probabilities and use a fixed recent context. The GPT-2 report (2019) uses the same autoregressive factorization and contrasts task-specific labeled datasets with learning from naturally occurring sequences.

Bengio’s model predicts one next word from a fixed recent context. Here, one aligned pair packages $T$ consecutive next-token targets while deriving every target from the same observed sequence order.

This contrast does not imply that early language models needed people to label every next word. The cited papers also do not prescribe this course’s fixed split, document boundaries, stride, or full-pair rule. Those are our data-handling choices: they preserve document identity and train/validation/test membership so later fitting code can select training documents only, and they make the number of examples reproducible.

Build complete pairs without flattening the corpus

CausalWindowConfig::new rejects a zero context length or stride. It also rejects a context length equal to usize::MAX, because $T+1$ would overflow. For a valid configuration, the policy counts candidate starts with a complete $T+1$ -token span and reports the suffix at the next start when it is too short for another pair:

Set the context, stride, count, and too-short-suffix policy rust/crates/llm-from-scratch/src/data.rs#causal-window-policy

/// Positive sizes that determine which candidate starts may produce pairs.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct CausalWindowConfig {
    context_length: usize,
    stride: usize,
    required_source_tokens: usize,
}

impl CausalWindowConfig {
    /// Validates the context length and distance between candidate starts.
    pub const fn new(
        context_length: usize,
        stride: usize,
    ) -> Result<Self, CausalWindowConfigError> {
        if context_length == 0 {
            return Err(CausalWindowConfigError::ZeroContextLength);
        }
        if stride == 0 {
            return Err(CausalWindowConfigError::ZeroStride);
        }
        let Some(required_source_tokens) = context_length.checked_add(1) else {
            return Err(CausalWindowConfigError::ContextLengthOverflow);
        };
        Ok(Self {
            context_length,
            stride,
            required_source_tokens,
        })
    }

    /// Returns the number of input IDs and target IDs in every emitted pair.
    pub const fn context_length(self) -> usize {
        self.context_length
    }

    /// Returns the distance between consecutive candidate starts.
    pub const fn stride(self) -> usize {
        self.stride
    }

    /// Returns the number of source IDs required to emit one shifted pair.
    pub const fn required_source_tokens(self) -> usize {
        self.required_source_tokens
    }

    /// Counts complete pairs without adding potentially huge indices.
    pub const fn window_count(self, document_length: usize) -> usize {
        if document_length < self.required_source_tokens {
            0
        } else {
            (document_length - self.required_source_tokens) / self.stride + 1
        }
    }

    /// Borrows every complete shifted pair selected inside one token slice.
    pub const fn windows<'a>(self, tokens: &'a [u32]) -> CausalWindows<'a> {
        CausalWindows {
            tokens,
            config: self,
            next_start: Some(0),
            remaining: self.window_count(tokens.len()),
        }
    }

    /// Returns the suffix at the first candidate start that cannot fill a pair.
    ///
    /// `None` means that the next candidate start lies at or beyond the end of
    /// the document. A returned suffix may overlap earlier complete pairs.
    pub fn incomplete_tail<'a>(self, tokens: &'a [u32]) -> Option<IncompleteTail<'a>> {
        let window_count = self.window_count(tokens.len());
        let start = if window_count == 0 {
            0
        } else {
            (window_count - 1)
                .checked_mul(self.stride)?
                .checked_add(self.stride)?
        };
        (start < tokens.len()).then(|| IncompleteTail {
            start,
            tokens: &tokens[start..],
        })
    }
}
/// A rejected configuration for causal-example construction.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum CausalWindowConfigError {
    /// An empty input and target would not teach a next-token relation.
    ZeroContextLength,
    /// `T + 1` cannot be represented, so the source requirement is undefined.
    ContextLengthOverflow,
    /// A zero stride would select the same start forever.
    ZeroStride,
}

impl fmt::Display for CausalWindowConfigError {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        formatter.write_str(match self {
            Self::ZeroContextLength => "context length must be positive",
            Self::ContextLengthOverflow => {
                "context length is too large to require one additional source token"
            }
            Self::ZeroStride => "stride must be positive",
        })
    }
}

impl Error for CausalWindowConfigError {}

CausalWindows then visits those starts. Its central operation borrows the input slice and the target slice one position to its right. Building a pair neither copies nor changes token IDs:

Borrow the input and its one-token-shifted target rust/crates/llm-from-scratch/src/data.rs#causal-window-iterator

/// One complete input/target pair borrowed from a single document.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct CausalWindow<'a> {
    start: usize,
    input: &'a [u32],
    target: &'a [u32],
}

impl<'a> CausalWindow<'a> {
    /// Returns the input's zero-based position inside its document.
    pub const fn start(self) -> usize {
        self.start
    }

    /// Returns exactly `context_length` source token IDs.
    pub const fn input(self) -> &'a [u32] {
        self.input
    }

    /// Returns the `T`-token source slice beginning one position after the input.
    pub const fn target(self) -> &'a [u32] {
        self.target
    }
}

/// The first selected suffix that is too short to emit a complete pair.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct IncompleteTail<'a> {
    start: usize,
    tokens: &'a [u32],
}

impl<'a> IncompleteTail<'a> {
    /// Returns the candidate position at which the too-short suffix begins.
    pub const fn start(self) -> usize {
        self.start
    }

    /// Returns the remaining source IDs; these may overlap earlier pairs.
    pub const fn tokens(self) -> &'a [u32] {
        self.tokens
    }
}

/// A repeatable, exact-size iterator over one document's complete pairs.
#[derive(Clone, Debug)]
pub struct CausalWindows<'a> {
    tokens: &'a [u32],
    config: CausalWindowConfig,
    next_start: Option<usize>,
    remaining: usize,
}

impl<'a> Iterator for CausalWindows<'a> {
    type Item = CausalWindow<'a>;

    fn next(&mut self) -> Option<Self::Item> {
        if self.remaining == 0 {
            return None;
        }

        let start = self.next_start?;
        let input_end = start.checked_add(self.config.context_length)?;
        let target_start = start.checked_add(1)?;
        let target_end = input_end.checked_add(1)?;
        let input = self.tokens.get(start..input_end)?;
        let target = self.tokens.get(target_start..target_end)?;

        self.remaining -= 1;
        self.next_start = if self.remaining == 0 {
            None
        } else {
            start.checked_add(self.config.stride)
        };
        Some(CausalWindow {
            start,
            input,
            target,
        })
    }

    fn size_hint(&self) -> (usize, Option<usize>) {
        (self.remaining, Some(self.remaining))
    }
}

impl ExactSizeIterator for CausalWindows<'_> {}
impl FusedIterator for CausalWindows<'_> {}

The iterator cannot discover document boundaries from token IDs alone: its input slice must contain exactly one document. The corpus wrapper below makes that intended traversal explicit by opening the iterator on an EncodedDocument.

The counting and suffix methods use checked arithmetic for extreme sizes. incomplete_tail is the API name for the suffix at the first candidate start that cannot form a pair. It returns None when that start is already at or beyond the document end. A returned suffix may overlap earlier complete pairs.

EncodedCorpusPartitions preserves each encoded sequence’s document ID and its train/validation/test membership. It encodes every document with the frozen Chapter 4 tokenizer and exposes documents only inside their original partition. It does not reuse the temporary IDs on which Chapter 3 learned merge rules, and it offers no flattened corpus view:

Keep encoded documents owned and separated by frozen partition rust/crates/llm-from-scratch/src/data.rs#partition-encoding

/// One owned token sequence with its stable document and partition identities.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct EncodedDocument {
    id: String,
    partition: Partition,
    token_ids: Vec<u32>,
}

impl EncodedDocument {
    /// Returns the stable source-document identity.
    pub fn id(&self) -> &str {
        &self.id
    }

    /// Returns the frozen partition role inherited from the split manifest.
    pub const fn partition(&self) -> Partition {
        self.partition
    }

    /// Returns the separately wrapped `[BOS, content..., EOS]` sequence.
    pub fn token_ids(&self) -> &[u32] {
        &self.token_ids
    }

    /// Opens a fresh borrowed pair iterator without consuming this document.
    pub fn windows(&self, config: CausalWindowConfig) -> CausalWindows<'_> {
        config.windows(&self.token_ids)
    }

    /// Reports the first candidate suffix that cannot fill a complete pair.
    pub fn incomplete_tail(&self, config: CausalWindowConfig) -> Option<IncompleteTail<'_>> {
        config.incomplete_tail(&self.token_ids)
    }
}

/// Encoded documents kept in three disjoint owned collections.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct EncodedCorpusPartitions {
    train: Vec<EncodedDocument>,
    validation: Vec<EncodedDocument>,
    test: Vec<EncodedDocument>,
}

impl EncodedCorpusPartitions {
    /// Applies one frozen tokenizer independently to every frozen source document.
    pub fn from_partitions(partitions: &CorpusPartitions<'_>, tokenizer: &BpeTokenizer) -> Self {
        let encode = |partition| {
            partitions
                .documents(partition)
                .iter()
                .map(|document| EncodedDocument {
                    id: document.id().to_owned(),
                    partition,
                    token_ids: tokenizer.encode_utf8_document(document.text()),
                })
                .collect()
        };

        Self {
            train: encode(Partition::Train),
            validation: encode(Partition::Validation),
            test: encode(Partition::Test),
        }
    }

    /// Returns only the separately encoded documents for one partition.
    pub fn documents(&self, partition: Partition) -> &[EncodedDocument] {
        match partition {
            Partition::Train => &self.train,
            Partition::Validation => &self.validation,
            Partition::Test => &self.test,
        }
    }
}

The same iterator is applied separately to each of the four short documents in the diagram below. Complete pairs and a possible too-short suffix are computed inside one document at a time, so no input or target can span two tapes.

The executable also loads the course’s bilingual corpus and fixed split manifest. As in Chapters 3–4, it trains BPE on the training partition only, obtains eight merge rules, freezes the tokenizer, and confirms that encoding preserves 8 training, 2 validation, and 2 test documents. It then opens the iterator on every encoded document and reports pair totals for each partition:

Run the worked pairs and preserve the frozen 8/2/2 corpus split rust/demos/ch05-autoregressive-examples/src/main.rs#chapter-output

use ch05_autoregressive_examples::hand_labeled_rows;
use llm_from_scratch::corpus::{Corpus, Partition, SplitManifest};
use llm_from_scratch::data::{CausalWindowConfig, EncodedCorpusPartitions};
use llm_from_scratch::tokenizer::bpe::BpeTokenizer;
use llm_from_scratch::tokenizer::bpe_trainer::BpeTrainer;

const CORPUS_JSON: &str = include_str!("../../../data/tiny-bilingual-corpus.json");
const SPLIT_MANIFEST: &str = include_str!("../../../data/splits.json");

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let rows = hand_labeled_rows();
    println!(
        "task-specific hand-labeled contrast: sentiment rows={}",
        rows.len()
    );
    for (index, row) in rows.iter().enumerate() {
        println!(
            "labeled row {index}: input={:?} label={}",
            row.input,
            row.label.as_str()
        );
    }

    let source = [0, 41, 42, 43, 44, 1];
    let config = CausalWindowConfig::new(3, 1)?;
    println!("source sequence for next-token targets: {source:?}");
    println!(
        "config: context={} stride={} required={}",
        config.context_length(),
        config.stride(),
        config.required_source_tokens()
    );
    println!("generated pairs: {}", config.window_count(source.len()));
    for window in config.windows(&source) {
        println!(
            "pair start={} input={:?} target={:?}",
            window.start(),
            window.input(),
            window.target()
        );
    }
    let tail = config
        .incomplete_tail(&source)
        .expect("worked source has one too-short suffix");
    println!(
        "next start too short: start={} tokens={:?} required={} emitted=false",
        tail.start(),
        tail.tokens(),
        config.required_source_tokens()
    );

    let short = [0, 61, 1];
    let short_tail = config
        .incomplete_tail(&short)
        .expect("short document leaves a too-short suffix");
    println!(
        "short document: {short:?} pairs={} suffix={:?}",
        config.window_count(short.len()),
        short_tail.tokens()
    );

    let corpus = Corpus::from_json(CORPUS_JSON)?;
    let manifest = SplitManifest::from_json(SPLIT_MANIFEST)?;
    let partitions = manifest.partition(&corpus)?;
    let training = BpeTrainer::new(8).train(&partitions)?;
    let tokenizer = BpeTokenizer::from_training(&training)?;
    let encoded = EncodedCorpusPartitions::from_partitions(&partitions, &tokenizer);
    println!(
        "frozen encoded documents: train={} validation={} test={}",
        encoded.documents(Partition::Train).len(),
        encoded.documents(Partition::Validation).len(),
        encoded.documents(Partition::Test).len()
    );
    let pair_counts = [Partition::Train, Partition::Validation, Partition::Test].map(|partition| {
        encoded
            .documents(partition)
            .iter()
            .map(|document| document.windows(config).count())
            .sum::<usize>()
    });
    println!(
        "encoded pairs at context=3 stride=1: train={} validation={} test={}",
        pair_counts[0], pair_counts[1], pair_counts[2]
    );

    println!("chapter 6 handoff: count each adjacent training-document transition once");
    Ok(())
}

Run the deterministic checks from the repository root:

cargo test --workspace --locked
cargo run --quiet --locked -p ch05-autoregressive-examples | diff -u rust/demos/ch05-autoregressive-examples/expected.txt -

The second command prints nothing when the executable exactly matches the recorded output. Tests cover the shifted-slice rule, invalid sizes, empty token slices, empty-content wrapped documents, exact fits, too-short suffixes, document and partition separation, and byte-exact decoding with the frozen tokenizer.

Inspect shifts and hard boundaries on separate token tapes

Build aligned next-token pairs one document at a time

The diagram shows four short encoded documents. In every complete pair, the target begins one source position after the input; no pair crosses a document or partition boundary.

Context length: 3
Stride: 1
Source tokens required: 4

Input context Each target lies one source position to the right Next-token targets

Only spans containing all required source tokens become pairs.

Pair construction restarts at every document or partition boundary: no pair crosses the boundary, and no shift arrow joins separate tapes.

At each shown start, too few tokens remain for a new pair, although those tokens may already occur in earlier complete pairs.

BOS — document beginning EOS — document ending

Hard boundary

Training partition

Document train-a

Wrapped source tokens
BOS:0, 41, 42, 43, 44, EOS:1
1. Complete pair #0 Candidate start: 0
  
  0, 41, 42
  
  41, 42, 43
2. Complete pair #1 Candidate start: 1
  
  41, 42, 43
  
  42, 43, 44
3. Complete pair #2 Candidate start: 2
  
  42, 43, 44
  
  43, 44, 1
Too few tokens for another pair
No new pair

Candidate start: 3 · Source tokens required: 4
43, 44, 1
Document train-b

Wrapped source tokens
BOS:0, 51, 52, EOS:1
1. Complete pair #0 Candidate start: 0
  
  0, 51, 52
  
  51, 52, 1
Too few tokens for another pair
No new pair

Candidate start: 1 · Source tokens required: 4
51, 52, 1

Hard boundary

Validation partition

Document validation-a

Wrapped source tokens
BOS:0, 61, 62, 63, EOS:1
1. Complete pair #0 Candidate start: 0
  
  0, 61, 62
  
  61, 62, 63
2. Complete pair #1 Candidate start: 1
  
  61, 62, 63
  
  62, 63, 1
Too few tokens for another pair
No new pair

Candidate start: 2 · Source tokens required: 4
62, 63, 1

Hard boundary

Test partition

Document test-a

Wrapped source tokens
BOS:0, 71, EOS:1
Too few tokens for another pair
No new pair

Candidate start: 0 · Source tokens required: 4
0, 71, 1

The four tapes are small examples, not an enumeration of all twelve corpus documents. Held-out pairs demonstrate the same construction, but only pairs from the training partition may be used to fit model parameters.

Read each input row with its target beneath it: the target begins one source position later. No arrow crosses the double line at a document or partition boundary; pair construction restarts on the next tape. A dotted outline marks a suffix that is too short for another pair. Its tokens are not deleted and may already appear in earlier complete pairs.

Predict the pairs, then check them

List every input and target for [0,41,42,43,44,1] with $T=3, S=1$ .
Repeat with $S=2$ . Which starts produce pairs, which otherwise valid start is skipped, and what remains at the next candidate start?
For [0,61,1] with $T=3$ , predict the pair count and suffix. Why would padding it define a different target policy?
For [0,1] with $T=1$ , decide whether BOS can predict EOS.
Two documents are [0,10,1] and [0,20,1]. With $T=2$ , could [10,1] -> [1,0] ever be emitted?
With $T=3, S=5$ and a four-token document, should an empty suffix be reported after the exact-fit pair?
Explain how tokens in a suffix that is too short for a new pair can still occur in earlier complete pairs.
Explain why Chapter 6 should not count transitions by iterating all overlapping pairs.
Does shifting the stored target by one position prevent a model from reading later input positions?

Check your predictions

Starts 0, 1, and 2 produce the three pairs in the worked table. At start 3, the suffix is too short for another pair.
Starts 0 and 2 produce pairs. Start 1 is skipped by the stride, not rejected for being short. The next candidate start is 4, where [44,1] remains—too few tokens for a pair at $T=3$ .
It emits zero pairs. The suffix begins at 0 and contains [0,61,1]; it is not padded. Padding would invent extra input positions and would require a mask plus an explicit rule for which padded targets count, so it is a different data policy.
Yes. The complete two-token source span gives [0] -> [1].
No. Such a pair exists only in an incorrectly flattened slice: its final prediction would be EOS -> BOS across documents. The first document’s slice ends after EOS, and construction starts a new iterator on [0,20,1].
One exact-fit pair is produced at start 0. The next candidate start is 5, beyond the document, so incomplete_tail returns None; no phantom empty suffix appears.
No new pair can begin in that suffix, but its tokens may already occur in complete pairs that began earlier. In the worked example, [43,44,1] already appears across earlier inputs and targets.
Overlap repeats the same adjacent transition in several views. Chapter 6 must scan each original training document once so every transition contributes once.
No. The shifted arrays define the correct targets. The model must separately enforce causal visibility through left-to-right recurrent state or a causal self-attention mask.

To test one policy change, first predict the changed starts, pairs, and suffix for the six-token source. Then temporarily change only the stride in CausalWindowConfig::new(3, 1) in rust/demos/ch05-autoregressive-examples/src/main.rs from 1 to 2 and run cargo run --quiet --locked -p ch05-autoregressive-examples. Alternatively, change only the context length from 3 to 2. The full-corpus pair totals will also change, but you are not expected to calculate those by hand. Compare the worked example’s pairs and suffix with your prediction, then restore CausalWindowConfig::new(3, 1) to return the demo to its baseline configuration.

Misconception check: “A suffix that is too short to begin another pair is discarded from the dataset.” False. No new pair begins there, but its tokens may already occur in complete pairs that began earlier.

Connect the pairs to the decoder’s task

Each complete pair states the decoder’s task position by position. At position $i$ , predict $y_i$ from $x_0$ through $x_i$ ; causal computation must hide $x_{i+1}$ and later inputs. BOS can anchor an initial prediction, and EOS can be the final target. Neither is padding.

The shifted arrays identify what to predict at each position; they do not prevent a model from inspecting later input positions. A left-to-right recurrent model enforces that limit through its sequential state. A decoder using self-attention needs a causal mask that blocks access to later positions.

Later chapters will replace integer counts with tensors, embeddings, attention, and learned parameters that produce logits, but this alignment remains the target relation. Chapter 6 takes one deliberate detour through an inspectable bigram baseline. It reads the original encoded training documents, not the overlapping pairs, so each adjacent transition is counted exactly once and validation and test documents do not influence the fitted transition counts.

Building autoregressive input–target pairs

Predict all three input–target pairs before running the code

Express the one-token shift with slices

What each symbol means

Derive next-token targets from the sequence itself

Build complete pairs without flattening the corpus

Inspect shifts and hard boundaries on separate token tapes

Build aligned next-token pairs one document at a time

Training partition

Document `train-a`

Wrapped source tokens

Too few tokens for another pair

Document `train-b`

Wrapped source tokens

Too few tokens for another pair

Validation partition

Document `validation-a`

Wrapped source tokens

Too few tokens for another pair

Test partition

Document `test-a`

Wrapped source tokens

Too few tokens for another pair

Predict the pairs, then check them

Connect the pairs to the decoder’s task

Predict all three input–target pairs before running the code

Express the one-token shift with slices

What each symbol means

Derive next-token targets from the sequence itself

Build complete pairs without flattening the corpus

Inspect shifts and hard boundaries on separate token tapes

Build aligned next-token pairs one document at a time

Rules shown in the diagram

Wrapped source tokens

Wrapped source tokens

Wrapped source tokens

Wrapped source tokens

Predict the pairs, then check them

Connect the pairs to the decoder’s task