From tensor coordinates to one flat buffer

Predict one address before flattening the picture

A tensor gives an $n$-dimensional meaning to a collection of scalar values. Start with this shape:

[2, 2, 3]

It has three axes, so its rank is 3. The extents say that axis 0 has two slices, each slice has two rows along axis 1, and each row has three positions along axis 2. Here are the logical values:

slice 0: [[10, 11, 12], [20, 21, 22]]
slice 1: [[30, 31, 32], [40, 41, 42]]

This tensor implementation does not store nested row objects. It owns these values in one flat Vec<f64>:

[10, 11, 12, 20, 21, 22, 30, 31, 32, 40, 41, 42]

Before reading further, make two predictions:

1. At which zero-based flat position does coordinate [1,0,2] land, and which value is stored there?
2. Can coordinate [1,2,0] select a value? If not, which axis fails first?

In row-major order, the last coordinate changes fastest. Moving one step on axis 2 advances by one flat element. A complete row contains three values, so moving one step on axis 1 advances by three. A complete slice contains two rows of three, so moving one step on axis 0 advances by six. Those distances are the element strides:

[6, 3, 1]

Now expand the first prediction:

The answer contains two different results. Offset 8 is an integer position in the flat buffer. Value 32 is the f64 stored at that position. Calling both of them “the index” hides the operation that connects coordinates to storage.

The second prediction fails before any buffer access. Axis 1 has extent 2, so its only valid indices are 0 and 1. Coordinate [1,2,0] supplies index 2 at that axis. The error reports axis 1, index 2, and size 2 rather than allowing the arithmetic to wander into another logical row.

Turn each axis movement into one stride term

For a rank-$d$ tensor, the coordinate-to-offset rule is:

The coordinate contains one zero-based index $i_k$ for every axis $k$. Its element stride $s_k$ tells how many positions the flat offset moves when that index increases by one and all other indices stay fixed. Row-major strides are derived from right to left. The last stride is one; each earlier stride is the checked product of the extents to its right.

For shape [2,2,3], the suffix to the right of axis $0$ has $2\cdot3=6$ elements, the suffix to the right of axis $1$ has $3$, and nothing follows axis $2$, giving [6,3,1]. The stride unit is f64 elements, not bytes.

The formula assumes a valid coordinate. The implementation first requires exactly $d$ indices, then visits axes from first to last, checking $0 \le i_k < \operatorname{shape}[k]$ before adding that axis’s term. For [1,0,2], the terms are 6, 0, and 2, so the offset is 8.

Two edge shapes make the product convention precise. Shape [] has rank zero and an empty stride list, but it represents one scalar value: the empty coordinate maps to offset 0. Shape [2,0,3] has strides [0,3,1] and zero values. The zero extent makes the tensor empty, yet the constructor still derives and checks the complete suffix-stride chain.

Locate every symbol in the tensor

Shape entries and strides answer different questions. shape[k] is how many positions exist on axis $k$; $s_k$ is how far one step on that axis moves through the flat buffer. In the fixture, axis 0 has extent 2 but stride 6, and axis 1 has extent 2 but stride 3.

The formula has no terms when $d=0$, so its sum is zero. That gives the scalar’s empty coordinate offset 0; the constructor’s separate shape invariant gives the scalar one stored value. For an axis with extent zero, there is no valid $i_k$, so an empty tensor has no valid full coordinate even though its strides are well defined.

From bigram counts to learned matrices and attention tensors

The Chapter 6 bigram gives each current-token and next-token pair its own count and uses only one token of context, so it cannot share evidence through learned word similarity.

The two primary checkpoints are Bengio et al., A Neural Probabilistic Language Model and Vaswani et al., Attention Is All You Need.

Bengio et al. describe n-gram models as short-context conditional-probability tables that do not use word similarity, then define a neural language model with a vocabulary-size-by-feature-width matrix $C$ of learned word features and neural parameter matrices for next-word prediction. Vaswani et al. later pack simultaneous queries, keys, and values into matrices $Q$, $K$, and $V$ and use learned projections to run multiple attention heads in parallel before concatenating their outputs.

Bengio’s model still predicts from a fixed window of preceding words. For the toy dimensions in this chapter, $C$ has shape $[|V|,m]=[5,3]$ and stores three learned features for each of five vocabulary items. A two-word context gives the hidden mapping $H$ shape $[h,2m]=[4,6]$, and $U$ shape $[|V|,h]=[5,4]$ maps four hidden activations toward five next-word scores. These are small shape stand-ins, not the paper’s experiment sizes or a complete list of its parameters.

For the Transformer checkpoint, each one-head $Q$, $K$, and $V$ stand-in has shape $[\text{tokens},\text{head width}]=[2,3]$. The local $Q$ stack has shape $[\text{heads},\text{tokens},\text{head width}]=[2,2,3]$, making a separate head axis visible. The paper supports the matrix computation and parallel-head claims; it does not require that particular axis order.

Explicit tensor shapes let this course represent embeddings, learned weights, activations, and attention intermediates in the cumulative decoder; the single contiguous row-major buffer is a local implementation policy, not a requirement of either paper.

The runnable Rust contrast uses the same Tensor abstraction for all seven model-shaped stand-ins and prints their shapes, strides, and element counts. The zero values are inert placeholders: this chapter exposes a storage consequence of the model mathematics without implementing either architecture’s operations.

Make shape validity and indexing one checked responsibility

Tensor::from_vec accepts an owned Vec<usize> shape and an owned Vec<f64> data buffer. It derives row-major strides from right to left with checked multiplication, then requires the data length to equal the checked element count. The empty shape starts from the multiplicative identity, so it needs exactly one value. Any zero extent produces a zero element count.

The implementation does not inspect, clamp, or normalize data values. A tensor may store finite numbers, either infinity, NaNs, signed zero, or any other f64 bit pattern. The invariant concerns storage length and coordinate meaning, not the numerical interpretation that later operations will give the values.

pub(crate) fn checked_row_major_layout(
    shape: &[usize],
) -> Result<(Vec<usize>, usize), TensorError> {
    if shape.is_empty() {
        return Ok((Vec::new(), 1));
    }

    let mut strides = vec![1; shape.len()];
    for axis in (0..shape.len() - 1).rev() {
        strides[axis] = shape[axis + 1]
            .checked_mul(strides[axis + 1])
            .ok_or(TensorError::ShapeOverflow)?;
    }

    let element_count = shape[0]
        .checked_mul(strides[0])
        .ok_or(TensorError::ShapeOverflow)?;
    Ok((strides, element_count))
}

pub(crate) fn checked_offset(
    shape: &[usize],
    strides: &[usize],
    base_offset: usize,
    coordinate: &[usize],
) -> Result<usize, TensorError> {
    debug_assert_eq!(shape.len(), strides.len());

    if coordinate.len() != shape.len() {
        return Err(TensorError::RankMismatch {
            expected: shape.len(),
            actual: coordinate.len(),
        });
    }

    let mut offset = base_offset;
    for (axis, ((&index, &dimension), &stride)) in
        coordinate.iter().zip(shape).zip(strides).enumerate()
    {
        if index >= dimension {
            return Err(TensorError::IndexOutOfBounds {
                axis,
                index,
                dimension,
            });
        }
        let contribution = index
            .checked_mul(stride)
            .ok_or(TensorError::ShapeOverflow)?;
        offset = offset
            .checked_add(contribution)
            .ok_or(TensorError::ShapeOverflow)?;
    }
    Ok(offset)
}

impl Tensor {
    /// Builds a tensor after checking its row-major layout and buffer length.
    pub fn from_vec(shape: Vec<usize>, data: Vec<f64>) -> Result<Self, TensorError> {
        let (strides, expected) = checked_row_major_layout(&shape)?;
        let actual = data.len();
        if actual != expected {
            return Err(TensorError::DataLengthMismatch { expected, actual });
        }

        Ok(Self {
            data,
            shape,
            strides,
        })
    }

    /// Returns the number of logical axes.
    pub fn rank(&self) -> usize {
        self.shape.len()
    }

    /// Returns the extent of every axis.
    pub fn shape(&self) -> &[usize] {
        &self.shape
    }

    /// Returns the row-major suffix-product stride for every axis.
    pub fn strides(&self) -> &[usize] {
        &self.strides
    }

    /// Returns the number of stored values.
    pub fn len(&self) -> usize {
        self.data.len()
    }

    /// Reports whether the flat buffer stores no values.
    pub fn is_empty(&self) -> bool {
        self.data.is_empty()
    }

    /// Borrows the contiguous value buffer.
    pub fn as_slice(&self) -> &[f64] {
        &self.data
    }

    /// Mutably borrows the contiguous value buffer without changing its length.
    pub fn as_mut_slice(&mut self) -> &mut [f64] {
        &mut self.data
    }

    /// Consumes the tensor and returns its contiguous value buffer.
    pub fn into_vec(self) -> Vec<f64> {
        self.data
    }

The second region owns coordinate validation. offset first compares coordinate length with rank. It then visits axes from left to right and reports the first index that is not smaller than its extent. get and get_mut both call this same operation, so reading and mutation cannot drift into separate indexing rules.

    /// Maps one in-bounds coordinate to its row-major flat-buffer offset.
    pub fn offset(&self, coordinate: &[usize]) -> Result<usize, TensorError> {
        checked_offset(&self.shape, &self.strides, 0, coordinate)
    }

    /// Borrows the value at one checked coordinate.
    pub fn get(&self, coordinate: &[usize]) -> Result<&f64, TensorError> {
        let offset = self.offset(coordinate)?;
        Ok(&self.data[offset])
    }

    /// Mutably borrows the value at one checked coordinate.
    pub fn get_mut(&mut self, coordinate: &[usize]) -> Result<&mut f64, TensorError> {
        let offset = self.offset(coordinate)?;
        Ok(&mut self.data[offset])
    }

The public accessors expose the invariants without permitting shape and stride metadata to diverge. rank, shape, strides, len, is_empty, and as_slice inspect the tensor. as_mut_slice can change values but cannot resize the buffer; into_vec consumes the tensor and returns its owned values.

TensorError separates four failures. ShapeOverflow means a required usize product could not be represented. DataLengthMismatch { expected, actual } means the product was valid but the supplied buffer had the wrong length. RankMismatch { expected, actual } is checked before axes, and IndexOutOfBounds { axis, index, dimension } names the first invalid axis. The methods return these errors through Result; ordinary invalid input does not panic.

The demo’s first library region carries the model-history progression through one concrete type. It constructs tiny $C$, $H$, and $U$ parameter stand-ins, one-head $Q$, $K$, and $V$ activation stand-ins, and a local stacked-head $Q$ tensor. Every value is zero because only shape, strides, and element count matter here; later chapters own the actual model operations.

/// Builds tiny shape-only stand-ins for parameters and activations from the LLM history.
pub fn llm_shape_history_fixture() -> Result<Vec<(&'static str, Tensor)>, TensorError> {
    Ok(vec![
        ("toy Bengio C", Tensor::from_vec(vec![5, 3], vec![0.0; 15])?),
        ("toy Bengio H", Tensor::from_vec(vec![4, 6], vec![0.0; 24])?),
        ("toy Bengio U", Tensor::from_vec(vec![5, 4], vec![0.0; 20])?),
        (
            "toy Transformer Q (one head)",
            Tensor::from_vec(vec![2, 3], vec![0.0; 6])?,
        ),
        (
            "toy Transformer K (one head)",
            Tensor::from_vec(vec![2, 3], vec![0.0; 6])?,
        ),
        (
            "toy Transformer V (one head)",
            Tensor::from_vec(vec![2, 3], vec![0.0; 6])?,
        ),
        (
            "toy Transformer Q head stack",
            Tensor::from_vec(vec![2, 2, 3], vec![0.0; 12])?,
        ),
    ])
}

The second library region owns the chapter’s shared fixture. The learner program and diagram trace both construct the same shape and value sequence through this function, so the code and visual explanation use one tensor definition.

/// Reconstructs the immutable, contiguous tensor used throughout the chapter.
pub fn frozen_tensor_fixture() -> Result<Tensor, TensorError> {
    Tensor::from_vec(FROZEN_SHAPE.to_vec(), FROZEN_VALUES.to_vec())
}

The following main.rs excerpt first constructs the model-history shape fixture. It then computes the checked lookup [1,0,2], mutates [0,1,1] to 99, constructs a scalar and an empty tensor, and captures rank, bounds, and overflow errors. The code immediately after the displayed region prints those values. Its final output line makes the next boundary explicit: Chapter 9 will add a new shape, stride, and base-offset interpretation while retaining the same storage.

    let llm_shapes = llm_shape_history_fixture()?;

    let mut tensor = frozen_tensor_fixture()?;
    let selected_offset = tensor.offset(&SELECTED_COORDINATE)?;
    let selected_value = *tensor.get(&SELECTED_COORDINATE)?;
    *tensor.get_mut(&[0, 1, 1])? = 99.0;

    let scalar = Tensor::from_vec(vec![], vec![7.0])?;
    let scalar_offset = scalar.offset(&[])?;
    let scalar_value = scalar.get(&[])?;
    let empty = Tensor::from_vec(vec![2, 0, 3], vec![])?;

    let rank_error = tensor.offset(&[1, 0]).unwrap_err();
    let bounds_error = tensor.offset(&INVALID_COORDINATE).unwrap_err();
    let overflow_error = Tensor::from_vec(vec![usize::MAX, 2], vec![]).unwrap_err();

The separate diagram producer asks the frozen Tensor for its shape, strides, length, checked offset, value, and out-of-bounds error. Its serialization code then records the two logical slices and each axis contribution. Together, those records expose one checked coordinate, its stride terms and selected value, and one coordinate rejected before access.

    let tensor = frozen_tensor_fixture()?;
    let offset = tensor.offset(&SELECTED_COORDINATE)?;
    let value = tensor.get(&SELECTED_COORDINATE)?;
    let bounds_error = match tensor.offset(&INVALID_COORDINATE) {
        Err(error) => error,
        Ok(_) => {
            return Err(TensorError::IndexOutOfBounds {
                axis: 1,
                index: 2,
                dimension: 2,
            });
        }
    };
    let shape = usize_csv(tensor.shape());
    let strides = usize_csv(tensor.strides());
    let buffer = value_csv(tensor.as_slice());
    let coordinate = usize_csv(&SELECTED_COORDINATE);
    let slice0 = format!(
        "SLICE axis0=0 row0={} row1={}",
        row_csv(&tensor, 0, 0)?,
        row_csv(&tensor, 0, 1)?
    );
    let slice1 = format!(
        "SLICE axis0=1 row0={} row1={}",
        row_csv(&tensor, 1, 0)?,
        row_csv(&tensor, 1, 1)?
    );
    let terms = SELECTED_COORDINATE
        .iter()
        .zip(tensor.strides())
        .enumerate()
        .map(|(axis, (&index, &stride))| {
            format!(
                "TERM axis={axis} index={index} stride={stride} contribution={}",
                index * stride
            )
        })
        .collect::<Vec<_>>();

The recorded trace preserves the exact decimal spelling and ordering emitted by the Rust example. Read alongside the Tensor output, it shows that the slices, offset, selected value, and bounds error all describe the same stored values.

Follow the coordinate through slices, arithmetic, and storage

Read the figure in four passes:

1. Locate slice 1, row 0, and the third position in that row.
2. Match coordinate [1,0,2] with strides [6,3,1] and inspect all three recorded contributions.
3. Find offset 8 in the flat buffer, then distinguish its position from value 32.0.
4. Compare valid axis-1 index 0 with rejected index 2 in coordinate [1,2,0].

The two slice tables and flat buffer contain the same twelve values from the Rust trace. They are different views in the explanatory sense only: Chapter 8 has not implemented a TensorView API. The stride calculation shows why advancing the outermost axis skips six elements, while advancing the last axis skips one.

The diamond and double border mark offset 8 without relying on color. Match that marker first to coordinate [1,0,2], then to value 32.0 in the flat buffer.

The bounds panel is part of the same lesson, not an afterthought. It shows axis 1, index 2, and size 2; the tensor rejects that coordinate before producing an offset or reading a value. A plausible arithmetic result for invalid indices would not make the access valid.

Predict each layout result before checking it

Work from right to left when deriving strides, but check coordinate rank and axes from left to right when validating access.

1. Derive the row-major element strides for shape [3,4,5].
2. For shape [2,2,3], strides [6,3,1], and the frozen buffer, compute every contribution for coordinate [1,0,2]. State its offset and value separately.
3. Predict the error for coordinate [1,2,0]. Which axis, index, and size must the error report?
4. For scalar shape [], state rank, strides, element count, and the only valid coordinate and offset.
5. For shape [2,0,3], derive the strides and element count. Why can no full coordinate be valid?
6. Compare two failed constructors: shape [2,2] with three data values, and shape [usize::MAX,2] with an empty data vector. Which error belongs to each?
7. Predict which flat offset changes when get_mut(&[0, 1, 1]) assigns 99, and write the resulting flat buffer.

Misconception check: strides are not shape extents. An extent counts valid positions on an axis; a stride measures the flat-buffer movement caused by one step on that axis. Row-major is also a convention, not an intrinsic law of tensors. A column-major tensor or an indirect ragged structure can represent multidimensional data with a different coordinate-to-storage rule.

Reuse one checked numeric container throughout the model

The course now has a Tensor that owns one contiguous Vec<f64>. Shape gives its $n$-dimensional interpretation, row-major strides map valid coordinates to deterministic offsets, and checked access prevents rank or bounds mistakes from becoming unrelated buffer reads. Later chapters can build embeddings, activations, weights, attention intermediates, logits, losses, and gradients on this same storage invariant.

Chapter 9 will reinterpret the same storage through views and axis transforms. That work may introduce a base offset and non-default stride interpretations, but it must not silently copy the buffer or change its values. Arithmetic, broadcasting, matrix multiplication, and differentiation remain separate later chapters; Chapter 8 contributes only the stable storage and checked indexing they will rely on.