18 · Content revision 7

Give token IDs trainable vectors

Build a trainable token table, validate public token IDs once, pass owned selectors through a private validated gather plan, and scatter-add repeated-token gradients.

Predict three selected rows and one shared gradient

Use this trainable table, whose four vocabulary rows each have width $2$ :

E=\begin{bmatrix} 10 & 11 \\ 20 & 21 \\ 30 & 31 \\ 40 & 41 \end{bmatrix}

Before running anything, read the IDs $[[2,1,2]]$ from left to right. The first and third positions both select row $2$ ; the middle position selects row $1$ . The predicted output is therefore $[[[30,31],[20,21],[30,31]]]$ , with shape $[1,3,2]$ . Repetition copies a value into two positions, but it does not create a second trainable row.

One-hot notation makes the selection visible. For example, ID $2$ corresponds to $[0,0,1,0]$ . Multiplying that indicator by $E$ leaves $[30,31]$ . The runnable contrast materializes those zeros only for this tiny explanation:

Multiply explicit one-hot rows by the tiny table as an algebraic baseline rust/demos/ch18-token-embeddings/src/lib.rs#one-hot-baseline

/// Materializes the tiny historical algebraic baseline for comparison only.
pub fn explicit_one_hot_product(table: &Tensor, token_ids: &[u32]) -> (Vec<Vec<u8>>, Vec<f64>) {
    assert_eq!(table.shape().len(), 2);
    let vocabulary_size = table.shape()[0];
    let width = table.shape()[1];
    let mut indicators = Vec::with_capacity(token_ids.len());
    let mut output = Vec::with_capacity(token_ids.len() * width);

    for &token_id in token_ids {
        let selected = usize::try_from(token_id).expect("u32 token ID must fit usize");
        assert!(selected < vocabulary_size);
        let mut one_hot = vec![0_u8; vocabulary_size];
        one_hot[selected] = 1;
        for feature in 0..width {
            let value = one_hot
                .iter()
                .enumerate()
                .map(|(row, &active)| f64::from(active) * table.as_slice()[row * width + feature])
                .sum();
            output.push(value);
        }
        indicators.push(one_hot);
    }
    (indicators, output)
}

The actual layer selects the row directly and verifies that both routes produce the same exact values:

Gather the repeated IDs and compare exact output values with the baseline rust/demos/ch18-token-embeddings/src/lib.rs#known-token-lookup

    let embedding = known_embedding();
    let output = embedding.forward(&TOKEN_IDS, &TOKEN_SHAPE)?;
    let (_, one_hot_output) = explicit_one_hot_product(&known_table(), &TOKEN_IDS);
    let one_hot_matches = output.value().as_slice() == one_hot_output;

Select forward, accumulate backward

The complete lookup and reverse rule is:

X_{b,t,:}=E_{z_{b,t},:},\quad \bar{E}_{i,:}=\sum_{(b,t):z_{b,t}=i}\bar{X}_{b,t,:}

The forward half copies one selected row into each output position. Let $L$ be the scalar loss. An overbar is reverse-mode shorthand: $\bar{X}_{b,t,:}=\partial L/\partial X_{b,t,:}$ is the gradient arriving at an output position, while $\bar{E}_{i,:}=\partial L/\partial E_{i,:}$ is the gradient of table row $i$ . The reverse half follows the forward selections in the opposite direction. If several positions selected the same row, their upstream vectors add feature by feature into that one shared parameter row.

For the declared seed $[[[1,0],[0,2],[3,4]]]$ , row $1$ receives $[0,2]$ and row $2$ receives $[1,0]+[3,4]=[4,4]$ . Unused rows $0$ and $3$ receive zero. Token IDs receive no gradient because they are discrete selectors, not tape operands.

Keep vocabulary axes separate from feature axes

$E$ is the trainable token table with shape $[V,d]$ .
$V$ is the vocabulary size and therefore the number of rows in $E$ .
$d$ is the embedding width and therefore the number of features in each row.
$z_{b,t}$ is the integer token ID at batch index $b$ and sequence position $t$ .
$b$ identifies one batch item; $t$ identifies one position in its sequence.
$:$ means every coordinate on the final feature axis.
$X_{b,t,:}$ is the selected width- $d$ vector at position $(b,t)$ .
$\bar{X}_{b,t,:}=\partial L/\partial X_{b,t,:}$ is the upstream gradient vector arriving at that output position.
$\bar{E}_{i,:}=\partial L/\partial E_{i,:}$ is the table-row gradient after contributions from every matching position have been accumulated.
$i$ is one vocabulary-row index.
The sum visits every $(b,t)$ whose $z_{b,t}$ equals $i$ .

The numeric distance between IDs is not a semantic distance. IDs $1$ and $2$ are adjacent only in the tokenizer’s numbering; training determines whether their learned vectors become similar.

From sparse identity to the vector entrance of a Transformer

A sparse one-hot word representation assigns one coordinate to each vocabulary item but expresses no graded similarity between words; explicitly carrying that vocabulary-wide vector also wastes work when only one row is needed.

Bengio et al., A Neural Probabilistic Language Model: Bengio et al. represent the mapping from a vocabulary word index to distributed features as a trainable matrix with one row per vocabulary item and one column per learned feature, share it across context positions, and learn it jointly with next-word prediction.

Bengio et al. learn a shared dense word-feature table jointly with a neural next-word model. The Transformer retains learned token embeddings for subword tokens, then adds positional information before its stacked attention and feed-forward computations.

Vaswani et al., Attention Is All You Need: Vaswani et al. use learned embeddings whose width matches the model width for BPE or word-piece tokens and add positional encodings before the Transformer stack; their embedding forward scaling is separate from parameter initialization.

The decoder’s token IDs enter the numeric model by selecting rows from one trainable vocabulary-by-feature table. Repeated IDs share the same parameter row, so their reverse contributions add; positional information, embedding forward scaling, attention, and output-weight tying remain later concerns.

One-hot vectors make token identity explicit but carry a vocabulary-sized field of zeros. Learned dense word features let neural language models share statistical strength, and Transformers keep learned token embeddings as the numeric entrance to deeper sequence computation. The algebraic one-hot identity explains direct row lookup, while the shared trainable row explains why repeated-token gradients add.

The essential progression is from sparse identity codes to learned vectors that are shared wherever the same token occurs. A concrete implementation may choose different integer types, storage layouts, initializers, and error conventions without changing the lookup or reverse-mode equations. The one-hot product is an explanatory identity, not a claim that either paper stores those sparse vectors.

Validate token IDs once, then reuse the gather rule

The layer reports table, token-shape, count, bounds, conversion-allocation, and delegated autodiff failures without partial output:

Keep embedding construction and selector failures typed and deterministic rust/crates/llm-from-scratch/src/nn/embedding.rs#embedding-errors

/// A rejected embedding table, token layout, selector, or delegated operation.
#[derive(Clone, Debug, PartialEq)]
pub enum EmbeddingError {
    Initialization(InitializationError),
    Autodiff(TensorAutodiffError),
    TableRank {
        rank: usize,
    },
    EmptyVocabulary,
    ZeroEmbeddingWidth,
    TokenShape(TensorError),
    TokenCountMismatch {
        expected: usize,
        actual: usize,
    },
    TokenIdOutOfBounds {
        position: usize,
        id: u32,
        vocabulary_size: usize,
    },
    IndexAllocationFailed {
        elements: usize,
    },
}

impl fmt::Display for EmbeddingError {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::Initialization(error) => error.fmt(formatter),
            Self::Autodiff(error) => error.fmt(formatter),
            Self::TableRank { rank } => {
                write!(
                    formatter,
                    "embedding table must have rank two, got rank {rank}"
                )
            }
            Self::EmptyVocabulary => {
                formatter.write_str("embedding vocabulary must contain at least one row")
            }
            Self::ZeroEmbeddingWidth => {
                formatter.write_str("embedding width must be greater than zero")
            }
            Self::TokenShape(error) => write!(formatter, "invalid token-ID shape: {error}"),
            Self::TokenCountMismatch { expected, actual } => write!(
                formatter,
                "token-ID shape needs {expected} IDs, but received {actual}"
            ),
            Self::TokenIdOutOfBounds {
                position,
                id,
                vocabulary_size,
            } => write!(
                formatter,
                "token ID {id} at flat position {position} is out of bounds for vocabulary size {vocabulary_size}"
            ),
            Self::IndexAllocationFailed { elements } => write!(
                formatter,
                "could not reserve {elements} converted embedding indices"
            ),
        }
    }
}

impl Error for EmbeddingError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        match self {
            Self::Initialization(error) => Some(error),
            Self::Autodiff(error) => Some(error),
            Self::TokenShape(error) => Some(error),
            _ => None,
        }
    }
}

impl From<InitializationError> for EmbeddingError {
    fn from(error: InitializationError) -> Self {
        Self::Initialization(error)
    }
}

impl From<TensorAutodiffError> for EmbeddingError {
    fn from(error: TensorAutodiffError) -> Self {
        Self::Autodiff(error)
    }
}

Public callers obtain an Embedding through new, from_parameter, or by cloning an already validated layer. The constructors establish a nonempty rank-two table, cloning preserves it, and private fields prevent callers from replacing it. The construction code records the table’s vocabulary size $V$ and embedding width $d$ once:

Construct one named rank-two table and retain its vocabulary and feature widths rust/crates/llm-from-scratch/src/nn/embedding.rs#embedding-layer

/// One named trainable `[vocabulary_size, embedding_width]` token table.
#[derive(Debug)]
pub struct Embedding {
    table: NamedParameter,
    vocabulary_size: usize,
    embedding_width: usize,
}

impl Clone for Embedding {
    /// Clones the layer handle while preserving the table's tape-leaf identity.
    fn clone(&self) -> Self {
        Self {
            table: self.table.clone(),
            vocabulary_size: self.vocabulary_size,
            embedding_width: self.embedding_width,
        }
    }
}

impl Embedding {
    /// Initializes one table transactionally with Chapter 17's shape-based policy.
    ///
    /// The complete parameter name is used as supplied. Validation checks the
    /// name before the vocabulary and width, and every error preserves `rng`.
    pub fn new(
        parameter_name: impl Into<String>,
        vocabulary_size: usize,
        embedding_width: usize,
        rng: &mut SplitMix64,
    ) -> Result<Self, EmbeddingError> {
        let mut trial = rng.clone();
        let table = NamedParameter::xavier_uniform(
            parameter_name,
            vocabulary_size,
            embedding_width,
            &mut trial,
        )
        .map_err(|error| match error {
            InitializationError::ZeroFanIn => EmbeddingError::EmptyVocabulary,
            InitializationError::ZeroFanOut => EmbeddingError::ZeroEmbeddingWidth,
            other => EmbeddingError::Initialization(other),
        })?;
        let embedding = Self::from_parameter(table)?;
        *rng = trial;
        Ok(embedding)
    }

    /// Gives embedding semantics to an existing named trainable rank-two table.
    pub fn from_parameter(table: NamedParameter) -> Result<Self, EmbeddingError> {
        let shape = table.tensor().shape();
        if shape.len() != 2 {
            return Err(EmbeddingError::TableRank { rank: shape.len() });
        }
        if shape[0] == 0 {
            return Err(EmbeddingError::EmptyVocabulary);
        }
        if shape[1] == 0 {
            return Err(EmbeddingError::ZeroEmbeddingWidth);
        }
        Ok(Self {
            table,
            vocabulary_size: shape[0],
            embedding_width: shape[1],
        })
    }
}

Embedding::forward is the public boundary for token IDs supplied as u32. Because construction has already established the table shape $[V,d]$ , the method performs these checks in a fixed order:

compute the number of positions described by token_shape, rejecting an invalid or overflowing shape;
require token_ids.len() to equal that position count; and
scan the IDs in flat order, rejecting the first u32 value that cannot name one of the table’s $V$ rows.

Only after all IDs pass does the method reserve an owned Vec<usize> and convert the selectors. After the tape establishes that the table operand is available, the method creates Chapter 16’s crate-private RowGatherPlan. The plan owns the converted selectors and their logical shape and derives the output shape. Its trusted constructor does not rescan table rank, selector count, or selector bounds. The constructor-or-clone invariant supplies the table rank, while this forward call has just established the selector shape, count, and bounds:

Check each raw token-ID fact once and pass owned selectors through the validated plan rust/crates/llm-from-scratch/src/nn/embedding.rs#embedding-forward-boundary

impl Embedding {
    /// Selects one table row per `u32` token ID and appends the feature axis.
    ///
    /// Token IDs remain integer selectors rather than differentiable operands.
    /// After this boundary validates and converts them, the shared gather kernel
    /// consumes the sealed facts without scanning the selectors again.
    pub fn forward(
        &self,
        token_ids: &[u32],
        token_shape: &[usize],
    ) -> Result<TensorValue, EmbeddingError> {
        let (_, expected) =
            checked_row_major_layout(token_shape).map_err(EmbeddingError::TokenShape)?;
        if token_ids.len() != expected {
            return Err(EmbeddingError::TokenCountMismatch {
                expected,
                actual: token_ids.len(),
            });
        }

        for (position, &id) in token_ids.iter().enumerate() {
            let valid = usize::try_from(id)
                .ok()
                .is_some_and(|index| index < self.vocabulary_size);
            if !valid {
                return Err(EmbeddingError::TokenIdOutOfBounds {
                    position,
                    id,
                    vocabulary_size: self.vocabulary_size,
                });
            }
        }

        let mut indices = Vec::new();
        indices
            .try_reserve_exact(expected)
            .map_err(|_| EmbeddingError::IndexAllocationFailed { elements: expected })?;
        for &id in token_ids {
            indices.push(usize::try_from(id).expect("validated u32 token ID must fit usize"));
        }
        self.table
            .tensor()
            .gather_rows_with_plan(move |table| {
                RowGatherPlan::from_validated_indices(table, indices, token_shape.to_vec())
            })
            .map_err(EmbeddingError::Autodiff)
    }
}

This trust is narrow. The public TensorValue::gather_rows method still checks table rank, selector shape, selector count, and the first out-of-range selector for callers that supply raw inputs. A valid plan also does not promise that memory is already available: allocating the output buffer can still fail. Once allocation succeeds, the shared kernel performs the row copies and saves the same plan facts for reverse scatter-add. The embedding layer therefore avoids a second selector-validation scan without duplicating either the lookup or the VJP. Only ownership and reuse of checked facts change; output values and shapes, failure precedence, the lookup equation, saved VJP facts, and repeated-row gradient addition stay the same.

The exact nonuniform reverse seed makes repeated accumulation observable:

Reverse through the repeated lookup and read the stored table gradient rust/demos/ch18-token-embeddings/src/lib.rs#repeated-token-gradient

    let upstream = Tensor::from_vec(vec![1, 3, 2], UPSTREAM_VALUES.to_vec())?;
    output.backward_with_seed(&upstream.view(), GraphRetention::Retain)?;
    let table_gradient = embedding
        .table()
        .tensor()
        .gradient_snapshot()
        .expect("trainable table stores its gradient");

Construction composes with Chapter 17’s shape-based matrix initializer by passing $V$ and $d$ as the table’s two dimensions. That is an explicit initialization convention, not something derived from embedding lookup or required by the Transformer paper. Another implementation can choose a different initializer without changing the forward or reverse equations. The same seed and request reproduce values, while separate constructions remain distinct leaves. Cloning one layer deliberately preserves its parameter identity:

Initialize the table reproducibly and check clone identity rust/demos/ch18-token-embeddings/src/lib.rs#initialized-token-embedding

    let mut first_rng = SplitMix64::from_seed(18);
    let mut second_rng = SplitMix64::from_seed(18);
    let initialized = Embedding::new("token_embedding.weight", 4, 2, &mut first_rng)?;
    let reproduced = Embedding::new("token_embedding.weight", 4, 2, &mut second_rng)?;
    let initialized_reproducible =
        *initialized.table().tensor().value() == *reproduced.table().tensor().value();
    let clone_same_node = initialized
        .table()
        .tensor()
        .is_same_node(initialized.clone().table().tensor());

The executable example also checks an empty ID shape and the first out-of-bounds ID:

Accept an empty token shape and reject the first out-of-range ID rust/demos/ch18-token-embeddings/src/lib.rs#embedding-edge-cases

    let empty_output = embedding.forward(&[], &[0])?.value_snapshot();
    let bounds_rejected = matches!(
        embedding.forward(&[4], &[1]),
        Err(EmbeddingError::TokenIdOutOfBounds {
            position: 0,
            id: 4,
            vocabulary_size: 4,
        })
    );

Its printed evidence contains the selected rows and accumulated table gradient. A scalar ID shape $[]$ produces shape $[2]$ ; empty leading shapes append the width without producing values. Shape and count errors are reported before the first invalid selector. An all-coordinate finite-difference probe with step $10^{-6}$ agrees with the analytic table gradient within absolute tolerance $2\times10^{-6}$ .

Print the selected vectors and the shared embedding-table gradient rust/demos/ch18-token-embeddings/src/main.rs#learner-token-embeddings-output

    let report = learner_report()?;

    println!(
        "table: {} shape={}",
        report.table_name,
        shape(&report.table_shape)
    );
    println!(
        "ids: shape={} values={}",
        shape(&report.token_shape),
        report
            .token_ids
            .iter()
            .map(u32::to_string)
            .collect::<Vec<_>>()
            .join(",")
    );
    println!(
        "output: shape={} values={}",
        shape(report.output.shape()),
        fixed_list(report.output.as_slice())
    );
    println!(
        "one-hot multiplication equals lookup: {}",
        report.one_hot_matches
    );
    println!(
        "upstream: shape={} values={}",
        shape(report.upstream.shape()),
        fixed_list(report.upstream.as_slice())
    );
    println!(
        "table gradient: shape={} values={}",
        shape(report.table_gradient.shape()),
        fixed_list(report.table_gradient.as_slice())
    );

Follow every selection back to its shared row

The forward half aligns each position with its token ID, one-hot identity, selected table row, and output vector. The reverse half aligns each upstream vector with the row that receives it. Row $2$ appears at two positions, so the diagram makes the difference visible: its value is copied to two outputs in the forward pass, while both gradient contributions add into one parameter row in the reverse pass.

Collect the worked example's table rows, selections, and accumulated gradients rust/demos/ch18-token-embeddings/src/diagram_trace.rs#token-embeddings-trace

pub fn render_trace() -> Result<String, Box<dyn Error>> {
    let embedding = known_embedding();
    let output = embedding.forward(&TOKEN_IDS, &TOKEN_SHAPE)?;
    let upstream = Tensor::from_vec(vec![1, 3, 2], UPSTREAM_VALUES.to_vec())?;
    output.backward_with_seed(&upstream.view(), GraphRetention::Retain)?;
    let gradient = embedding
        .table()
        .tensor()
        .gradient()
        .expect("trainable table stores its gradient");
    let (one_hot_rows, baseline) = explicit_one_hot_product(&known_table(), &TOKEN_IDS);
    assert_eq!(baseline, output.value().as_slice());
    let mut repeated_ids = TOKEN_IDS
        .iter()
        .copied()
        .filter(|candidate| TOKEN_IDS.iter().filter(|id| *id == candidate).count() > 1)
        .collect::<Vec<_>>();
    repeated_ids.sort_unstable();
    repeated_ids.dedup();
    assert_eq!(
        repeated_ids.len(),
        1,
        "fixture must contain one repeated token ID"
    );
    let repeated_id = repeated_ids[0];
    let repeated_positions = TOKEN_IDS
        .iter()
        .enumerate()
        .filter_map(|(position, id)| (*id == repeated_id).then_some(position))
        .collect::<Vec<_>>();

    let mut trace = String::new();
    writeln!(trace, "TRACE token-embeddings-v1 BEGIN")?;
    writeln!(
        trace,
        "FIXTURE name=known-table-repeated-id parameter={} vocabulary={} width={} table-shape={} id-shape={} output-shape={} upstream-shape={} gradient-shape={} accumulation=scatter-add",
        embedding.table().name(),
        embedding.vocabulary_size(),
        embedding.embedding_width(),
        shape(embedding.table().tensor().shape().as_slice()),
        shape(&TOKEN_SHAPE),
        shape(output.shape().as_slice()),
        shape(upstream.shape()),
        shape(gradient.shape()),
    )?;
    writeln!(
        trace,
        "IDS values={} repeated-id={repeated_id} repeated-flat-positions={}",
        token_list(&TOKEN_IDS),
        integer_list(&repeated_positions),
    )?;

    for row in 0..TABLE_SHAPE[0] {
        let uses = TOKEN_IDS
            .iter()
            .filter(|&&id| usize::try_from(id).ok() == Some(row))
            .count();
        let state = match uses {
            0 => "unused",
            1 => "selected-once",
            _ => "selected-repeated",
        };
        let start = row * embedding.embedding_width();
        writeln!(
            trace,
            "TABLE row={row} uses={uses} state={state} values={}",
            fixed_list(
                &embedding.table().tensor().value().as_slice()
                    [start..start + embedding.embedding_width()]
            )
        )?;
    }

    for (flat, (&id, one_hot)) in TOKEN_IDS.iter().zip(&one_hot_rows).enumerate() {
        let start = flat * embedding.embedding_width();
        let coordinate = row_major_coordinate(flat, &TOKEN_SHAPE);
        let uses = TOKEN_IDS
            .iter()
            .filter(|&&candidate| candidate == id)
            .count();
        writeln!(
            trace,
            "LOOKUP flat={flat} coordinate={} id={id} sharing={} one-hot={} selected-row={id} output={} upstream={}",
            integer_list(&coordinate),
            if uses > 1 {
                "repeated-row"
            } else {
                "single-row"
            },
            integer_list(
                &one_hot
                    .iter()
                    .map(|&value| usize::from(value))
                    .collect::<Vec<_>>()
            ),
            fixed_list(&output.value().as_slice()[start..start + embedding.embedding_width()]),
            fixed_list(&UPSTREAM_VALUES[start..start + embedding.embedding_width()])
        )?;
    }

    for row in 0..TABLE_SHAPE[0] {
        let positions = TOKEN_IDS
            .iter()
            .enumerate()
            .filter_map(|(position, &id)| {
                (usize::try_from(id).ok() == Some(row)).then_some(position)
            })
            .collect::<Vec<_>>();
        let positions_text = if positions.is_empty() {
            "none".to_owned()
        } else {
            integer_list(&positions)
        };
        let contributions_text = if positions.is_empty() {
            "none".to_owned()
        } else {
            positions
                .iter()
                .map(|position| {
                    let start = position * TABLE_SHAPE[1];
                    fixed_list(&UPSTREAM_VALUES[start..start + TABLE_SHAPE[1]])
                })
                .collect::<Vec<_>>()
                .join("|")
        };
        let start = row * embedding.embedding_width();
        let rule = match positions.len() {
            0 => "unused-zero",
            1 => "single-copy",
            _ => "repeated-sum",
        };
        writeln!(
            trace,
            "ROW-GRADIENT row={row} flat-positions={positions_text} contributions={contributions_text} rule={rule} accumulated={}",
            fixed_list(&gradient.as_slice()[start..start + embedding.embedding_width()])
        )?;
    }
    writeln!(trace, "TRACE token-embeddings-v1 END")?;
    Ok(trace)
}

Follow repeated token IDs through one shared table

Read the exact Rust-authored token IDs, table rows, one-hot lookup equivalence, output vectors, and reverse contributions that accumulate into the shared embedding table.

Named parameter: token_embedding.weight
Vocabulary rows: $4$
Embedding width: $2$
Table shape: $\left[4,2\right]$
Token-ID shape: $\left[1,3\right]$
Output shape: $\left[1,3,2\right]$
Table-gradient shape: $\left[4,2\right]$

Start with integer token IDs

IDs name rows and stay outside the differentiation tape. Repeating an ID reuses one parameter row.

Position	Token ID	Row status
$\left(0,0\right)$	$2$	Shares one row with another position
$\left(0,1\right)$	$1$	Selects one singly used row
$\left(0,2\right)$	$2$	Shares one row with another position

Keep one shared trainable table

Table row	Trainable vector	Selections	Row status
$E_{0,:}$	$\left[10.000000000000,11.000000000000\right]$	$0$	Unused row
$E_{1,:}$	$\left[20.000000000000,21.000000000000\right]$	$1$	Selected once
$E_{2,:}$	$\left[30.000000000000,31.000000000000\right]$	$2$	Shared by the repeated token ID
$E_{3,:}$	$\left[40.000000000000,41.000000000000\right]$	$0$	Unused row

Select rows without storing the zeros

The indicator exposes the algebra: one active coordinate selects one table row. The implementation performs direct lookup.

Position	Token ID	One-hot indicator	Equivalent product	Selected row	Output vector	Upstream gradient
$\left(0,0\right)$	$2$	$\left[0,0,1,0\right]$	$e_{2}E$	$E_{2,:}$	$\left[30.000000000000,31.000000000000\right]$	$\left[1.000000000000,0.000000000000\right]$
$\left(0,1\right)$	$1$	$\left[0,1,0,0\right]$	$e_{1}E$	$E_{1,:}$	$\left[20.000000000000,21.000000000000\right]$	$\left[0.000000000000,2.000000000000\right]$
$\left(0,2\right)$	$2$	$\left[0,0,1,0\right]$	$e_{2}E$	$E_{2,:}$	$\left[30.000000000000,31.000000000000\right]$	$\left[3.000000000000,4.000000000000\right]$

Return contributions to their shared rows

Reverse mode returns each upstream feature vector to its selected row. Only the repeated row receives two vectors and sums them.

Table row	Contributing flat positions	Feature-wise contributions	Reverse rule	Stored row gradient
$\bar E_{0,:}$	None	None	No selection; keep zero	$\left[0.000000000000,0.000000000000\right]$
$\bar E_{1,:}$	$1$	$\left[0.000000000000,2.000000000000\right]$	One selection; copy its contribution	$\left[0.000000000000,2.000000000000\right]$
$\bar E_{2,:}$	$0,2$	$\left[1.000000000000,0.000000000000\right]$ $+$ $\left[3.000000000000,4.000000000000\right]$	Repeated selections; add both contributions	$\left[4.000000000000,4.000000000000\right]$
$\bar E_{3,:}$	None	None	No selection; keep zero	$\left[0.000000000000,0.000000000000\right]$

Read the forward and reverse halves together. The repeated state is marked by words, a diamond, and a double rule rather than color alone, so row sharing stays visible independently of the color scheme.

Predict before checking the executable evidence

Predict all six output values for IDs $[[2,1,2]]$ .
Write the length-four one-hot row for ID $2$ and multiply it by $E$ .
Predict every table-gradient value for the declared upstream seed.
Explain why row $2$ receives a sum rather than two independent parameter gradients.
Predict the output shapes for ID shapes $[2,3]$ , $[]$ , and $[0]$ when $d=2$ .
Put an invalid token shape, an ID-count mismatch, and an out-of-range ID in reporting order; then find the first invalid ID in $[1,4,9]$ for a four-row table.
Explain why Embedding::forward may hand its checked IDs to a trusted plan while the public generic gather method must validate raw selectors. What allocation can still fail after the plan exists?
Predict whether cloning an embedding layer creates a new trainable leaf.
Explain why IDs receive no gradient and why adjacent ID numbers imply no semantic similarity.
Decide whether Bengio et al. require an explicitly materialized one-hot implementation.
Decide whether a repeated occurrence owns a new embedding parameter.

Check the predictions

The output is $[[[30,31],[20,21],[30,31]]]$ .
ID $2$ is $[0,0,1,0]$ , so multiplying it by $E$ yields $[30,31]$ .
The table gradient is $[[0,0],[0,2],[4,4],[0,0]]$ .
Both occurrences selected the same row- $2$ leaf, so $[1,0]$ and $[3,4]$ add into it.
The output shapes are $[2,3,2]$ , $[2]$ , and $[0,2]$ .
The method checks the token shape first, the exact ID count second, and the IDs in flat order third. ID $4$ at flat position $1$ is the first invalid selector because valid rows are $0$ through $3$ .
Every Embedding value contains the rank-two table established by its public constructors, and forward has just checked shape, count, and bounds before converting the IDs. The trusted plan owns those facts. A generic public caller has established none of them, so TensorValue::gather_rows must perform its complete validation. Allocating the output tensor’s value buffer can still fail after the plan exists.
No. A clone is another handle to the same named table leaf.
IDs are discrete selectors outside the tape, and their numeric assignment carries no geometry.
No. One-hot multiplication is used here only as an algebraic explanation.
No. Repetition reuses one table row; later position handling distinguishes occurrences without creating occurrence-specific embedding parameters.

Hand the final feature axis to a learned projection

The cumulative model can now turn token-ID tensors into differentiable feature tensors that append one embedding-width axis while keeping one shared named vocabulary-by-feature parameter. Chapter 19 treats that final embedding width as its input width and mixes features with a learned projection; lookup selects rows, while a linear layer combines coordinates.

This is the decoder’s numeric entrance, not yet a position-aware representation. Chapter 19 preserves all leading batch and sequence axes while applying a learned matrix to the final feature axis. Later chapters make attention position-aware by rotating projected queries and keys with RoPE; repeated occurrences still share one embedding-table row.