Shared views and explicit tensor copies

Predict two [3, 2] tensors before reading their values

Start with the same flat owner used by the example:

shape   = [2, 3]
strides = [3, 1]
storage = [10, 11, 12, 20, 21, 22]

The shape describes two rows and three columns. Chapter 8’s offset rule gives row-major source offsets [0,1,2,3,4,5]. Before reading the reveal, write the shape, strides, logical source-offset sequence, and nested values for each request:

1. view.reshape(&[3,2])
2. view.transpose(0,1)

Neither request may move a single f64. Commit to both reading orders before continuing.

Reveal: equal shapes, different logical orders

First, reshape the view to [3,2]. A reshape preserves the current logical row-major order, so the new strides are [2,1]. Its coordinates visit source offsets in the same sequence:

reshape values = [[10, 11], [12, 20], [21, 22]]
offsets        = [0, 1, 2, 3, 4, 5]

Second, transpose axes 0 and 1. The old column axis becomes the new row axis. Shape [2,3] becomes [3,2], and the matching strides [3,1] become [1,3]:

transpose values = [[10, 20], [11, 21], [12, 22]]
offsets          = [0, 3, 1, 4, 2, 5]

The two results have equal shapes and contain the same six scalars overall, yet the scalars occupy different coordinates. Neither changes the owner’s storage. For example, transposed coordinate [2,1] and source coordinate [1,2] both select offset 5 and therefore the same stored 22.0.

A view is not a copy. It is a checked interpretation containing shape, strides, and a base offset while the original Tensor remains the owner. This distinction is the chapter’s central prediction: same storage does not mean same logical order.

Separate reshape compatibility from axis permutation

Two compact rules separate reshape compatibility from permutation metadata:

The first equality is the element-count requirement for reshape. The source extents $n_k$ and requested extents $n'_j$ must have equal checked products. For the example, $2\cdot3=3\cdot2=6$, so [2,3] reshaped to [3,2] is size-compatible. Shape [4,2] needs eight values and fails before any view is created.

Equal products are not the whole rule in this course. Reshape also requires the source view to be row-major contiguous. That policy prevents reshape from silently allocating after a transpose or inner-axis slice. Call materialize when a new contiguous owner is intentionally required.

The last two equalities are for permutation. Output axis $k$ takes the extent and stride of source axis $\pi(k)$. For permutation [1,0], output axis $0$ takes source axis $1$, while output axis $1$ takes source axis $0$; shape and strides therefore become [3,2] and [1,3].

Account for every extent, axis, and stride

Reshape and permutation use different halves of the formula. A contiguous reshape derives fresh row-major strides from the requested shape. A permutation reorders existing shape entries and strides together. Unit-step slicing does neither: it retains the stride list, changes one extent, and adds $\operatorname{start}\cdot\operatorname{stride}[\operatorname{axis}]$ to the base offset.

Scalar shape [] has an empty product of one, one value, and an empty permutation. Any zero extent gives a product of zero. The implementation still checks the complete requested suffix-stride chain, so a later enormous extent cannot hide overflow behind an earlier zero.

From fixed context to split and merged attention heads

In Bengio et al.’s feed-forward configuration, learned feature vectors for a fixed number of preceding words are concatenated into one vector $x$ and used to predict the next-word distribution. Its layout is fixed by the selected context width rather than exposing sequence and head axes for a growing causal prefix.

An earlier model is Bengio et al., A Neural Probabilistic Language Model. Later sources are Vaswani et al., Attention Is All You Need and OpenAI’s official GPT-2 model.py.

Vaswani et al. define attention on query, key, and value matrices, compute scaled products with transposed keys, and run learned projections in parallel heads whose outputs are concatenated. OpenAI’s GPT-2 model.py projects one tensor with batch, sequence, and feature axes into packed query, key, and value groups, splits and transposes them to a head axis, multiplies by the key tensor with its last two axes transposed, then transposes and merges heads.

Bengio’s fixed-context vector puts each selected word feature into a predetermined slot. Transformer attention instead retains a position axis in $Q$, $K$, and $V$. Its scaled dot-product contains $QK^{\mathsf T}$, so the key’s sequence and feature axes exchange roles at the multiplication boundary. Multiple projected heads add another axis before their outputs are concatenated.

GPT-2 makes those layout changes concrete. Its attention projection first produces three packed feature groups and splits them into $Q$, $K$, and $V$. Each group reshapes feature width into heads and per-head width, then transposes $[\text{batch},\text{sequence},\text{head},\text{width}]$ into $[\text{batch},\text{head},\text{sequence},\text{width}]$. Attention multiplies $Q$ by $K$ with $K$‘s final two axes transposed. The per-head result is transposed back before the head and feature widths are merged.

Those sources establish model calculations and logical axis transformations. They do not prescribe whether an implementation copies values, shares a buffer, uses row-major strides, or rejects implicit materialization.

Reshape, axis permutation, and transpose let this course express the logical split-head, key-transpose, and merge-head layouts used by decoder attention. Whether a merge can reshape without copying depends on the resulting view’s contiguity. Borrowed TensorView and explicit materialization are local implementation policies, not storage behavior claimed by the papers or GPT-2’s TensorFlow code.

Treat the frozen [2,3] example as a deliberately tiny one-head $K$ with two key positions and three features; batch and head axes are suppressed. The existing copying_transpose function eagerly creates an owned contiguous $K$ transpose with values [10,20,11,21,12,22]. The borrowed transpose instead retains the owner and uses shape [3,2], strides [1,3], and offsets [0,3,1,4,2,5] to read the same logical order. This prepares one transposed operand; it does not implement $Q$, matrix multiplication, scaling, masking, softmax, or attention.

/// Builds a transposed tensor by explicitly allocating and copying every value.
pub fn copying_transpose(source: &Tensor) -> Result<Tensor, TensorError> {
    let [rows, columns] = source.shape() else {
        return Err(TensorError::RankMismatch {
            expected: 2,
            actual: source.rank(),
        });
    };
    if source.is_empty() {
        return Tensor::from_vec(vec![*columns, *rows], Vec::new());
    }

    let mut copied = Vec::with_capacity(source.len());
    for column in 0..*columns {
        for row in 0..*rows {
            copied.push(*source.get(&[row, column])?);
        }
    }
    Tensor::from_vec(vec![*columns, *rows], copied)
}

Borrow the owner, transform metadata, copy only on command

The cumulative Tensor still owns all values. Tensor::view stores an immutable reference to that owner, so Rust encodes the lifetime and alias rule in the type. Shape, strides, base offset, and length are copied metadata; scalar values are not copied.

/// An immutable n-dimensional interpretation of storage owned by a [`Tensor`].
///
/// The view copies only shape and stride metadata. Rust keeps the source tensor
/// borrowed for the view's lifetime, so safe code cannot mutate the owner while
/// a subsequently used view still exists:
///
/// ```compile_fail
/// use llm_from_scratch::tensor::storage::Tensor;
///
/// let mut tensor = Tensor::from_vec(vec![2], vec![10.0, 20.0]).unwrap();
/// let view = tensor.view();
/// tensor.as_mut_slice()[0] = 99.0;
/// assert_eq!(*view.get(&[0]).unwrap(), 10.0);
/// ```
#[derive(Clone, Debug, PartialEq)]
pub struct TensorView<'a> {
    source: &'a Tensor,
    shape: Vec<usize>,
    strides: Vec<usize>,
    base_offset: usize,
    len: usize,
}

impl Tensor {
    /// Borrows this tensor as a row-major-contiguous view without copying values.
    pub fn view(&self) -> TensorView<'_> {
        TensorView {
            source: self,
            shape: self.shape().to_vec(),
            strides: self.strides().to_vec(),
            base_offset: 0,
            len: self.len(),
        }
    }
}

impl<'a> TensorView<'a> {
    /// Returns the number of logical axes.
    pub fn rank(&self) -> usize {
        self.shape.len()
    }

    /// Returns the extent of every logical axis.
    pub fn shape(&self) -> &[usize] {
        &self.shape
    }

    /// Returns the source-storage movement for each logical axis.
    pub fn strides(&self) -> &[usize] {
        &self.strides
    }

    /// Returns the source-storage offset of the view's logical origin.
    pub fn base_offset(&self) -> usize {
        self.base_offset
    }

    /// Returns the number of logical values in the view.
    pub fn len(&self) -> usize {
        self.len
    }

    /// Reports whether the view has no logical values.
    pub fn is_empty(&self) -> bool {
        self.len == 0
    }

    /// Reports whether logical row-major iteration visits one dense storage span.
    ///
    /// Singleton axes may carry any stride because they never advance. Scalars
    /// and empty views are contiguous by this chapter's explicit convention.
    pub fn is_contiguous(&self) -> bool {
        if self.is_empty() {
            return true;
        }

        let mut expected_stride = 1_usize;
        for (&dimension, &stride) in self.shape.iter().zip(&self.strides).rev() {
            if dimension > 1 && stride != expected_stride {
                return false;
            }
            expected_stride = expected_stride
                .checked_mul(dimension)
                .expect("a valid tensor view retains a checked shape");
        }
        true
    }

    /// Maps one checked logical coordinate to the owner's flat storage offset.
    pub fn storage_offset(&self, coordinate: &[usize]) -> Result<usize, TensorViewError> {
        checked_offset(&self.shape, &self.strides, self.base_offset, coordinate).map_err(Into::into)
    }

    /// Borrows the source value selected by one checked logical coordinate.
    pub fn get(&self, coordinate: &[usize]) -> Result<&'a f64, TensorViewError> {
        let offset = self.storage_offset(coordinate)?;
        let source: &'a [f64] = self.source.as_slice();
        Ok(&source[offset])
    }
}

If code creates a view, mutates its owner, and then uses the view again, Rust rejects the conflicting borrow. Multiple immutable views are safe. materialize creates an independent tensor that can be mutated on its own; the original owner becomes mutable again after the borrowed view’s last use.

Axis transforms preserve source identity. reshape checks requested-layout overflow first, element count second, and current contiguity third. transpose builds a complete axis list and swaps two entries. permute requires exactly one in-range occurrence of every source axis.

For a separately contiguous $Q$, $K$, or $V$ tensor, those rank-generic operations can express the logical split from $[\text{batch},\text{sequence},\text{features}]$ to $[\text{batch},\text{heads},\text{sequence},\text{head width}]$. Inverse-permuting a view derived from the original contiguous split can restore contiguity. A newly allocated contiguous attention result already ordered as $[\text{batch},\text{heads},\text{sequence},\text{head width}]$ generally becomes non-contiguous after permutation to $[\text{batch},\text{sequence},\text{heads},\text{head width}]$; it must then be materialized before the head dimensions can be merged by reshape. Contiguity, not the operation’s name, decides whether reshape may share storage. The small rank-two example isolates only the transpose primitive; complete attention comes later.

There is also an intentional contiguity boundary. If this course represented GPT-2’s packed projection as $[\text{batch},\text{sequence},3\cdot\text{features}]$, slicing one query, key, or value block from the last axis when $\text{batch}\cdot\text{sequence}>1$ would retain gaps between outer rows. That multi-position slice would need explicit materialization before reshape; the GPT-2 TensorFlow source does not determine an implementation’s borrow-or-copy policy.

    /// Reinterprets a row-major-contiguous view with a compatible shape.
    pub fn reshape(&self, shape: &[usize]) -> Result<Self, TensorViewError> {
        let (strides, requested) = checked_row_major_layout(shape)?;
        if requested != self.len {
            return Err(TensorViewError::ReshapeElementCountMismatch {
                current: self.len,
                requested,
            });
        }
        if !self.is_contiguous() {
            return Err(TensorViewError::NonContiguousReshape);
        }

        Ok(Self {
            source: self.source,
            shape: shape.to_vec(),
            strides,
            base_offset: self.base_offset,
            len: self.len,
        })
    }

    /// Swaps two logical axes without moving source values.
    pub fn transpose(&self, first: usize, second: usize) -> Result<Self, TensorViewError> {
        self.check_axis(first)?;
        self.check_axis(second)?;

        let mut axes = (0..self.rank()).collect::<Vec<_>>();
        axes.swap(first, second);
        self.permute(&axes)
    }

    /// Reorders axes so output axis `k` uses source axis `axes[k]`.
    pub fn permute(&self, axes: &[usize]) -> Result<Self, TensorViewError> {
        if axes.len() != self.rank() {
            return Err(TensorViewError::PermutationLengthMismatch {
                expected: self.rank(),
                actual: axes.len(),
            });
        }

        let mut seen = vec![false; self.rank()];
        for &axis in axes {
            self.check_axis(axis)?;
            if seen[axis] {
                return Err(TensorViewError::DuplicateAxis { axis });
            }
            seen[axis] = true;
        }

        Ok(Self {
            source: self.source,
            shape: axes.iter().map(|&axis| self.shape[axis]).collect(),
            strides: axes.iter().map(|&axis| self.strides[axis]).collect(),
            base_offset: self.base_offset,
            len: self.len,
        })
    }

Public coordinate lookup and internal traversal serve different inputs. storage_offset and get accept one coordinate supplied by their caller, so each call checks the coordinate rank and every axis bound before computing its storage offset. materialize instead visits every logical position of a TensorView that a safe constructor or checked transform has already produced.

One crate-private cursor checks its internal traversal plan once, then owns one state record per axis. For a nonempty view, that check proves the greatest reachable source offset lies inside the owner’s storage. An empty view has no reachable offset, so the cursor yields nothing without trying to validate or read its unused base offset. External callers cannot use this cursor to bypass the checked public methods.

The cursor starts at the base offset and advances the last logical axis first. When an axis reaches its extent, it rewinds that axis and carries to the preceding axis. The frozen slice therefore yields [1,2,4,5]; a scalar yields its base offset once; an empty view yields no offsets. A zero effective stride deliberately repeats offsets for the broadcasting rules in Chapter 10, while a singleton axis never advances even if its stored stride is large.

The cursor uses safe Rust, retains no tensor borrow, and allocates its axis state once. Its next method neither allocates a new coordinate vector nor repeats public rank and axis-bound checks for every value. Reading a yielded offset still uses Rust’s ordinary bounds-checked slice indexing.

/// Logical row-major traversal over a layout whose metadata was already checked.
///
/// The cursor owns one `O(rank)` axis-state vector and updates it in place. It
/// keeps no tensor borrow, so later kernels can use the same plumbing for source
/// reads or destination writes. It remains crate-private so arbitrary external
/// coordinates still enter through [`TensorView::storage_offset`] or
/// [`TensorView::get`].
#[derive(Clone, Copy, Debug)]
struct OffsetAxis {
    extent: usize,
    stride: usize,
    position: usize,
    rewind: usize,
}

#[derive(Debug)]
pub(crate) struct StridedOffsets {
    axes: Vec<OffsetAxis>,
    next_offset: usize,
    remaining: usize,
}

impl StridedOffsets {
    /// Checks one internal traversal plan and then owns its reusable axis state.
    pub(crate) fn checked(
        shape: &[usize],
        strides: &[usize],
        base_offset: usize,
        logical_len: usize,
        backing_len: usize,
    ) -> Option<Self> {
        if shape.len() != strides.len() {
            return None;
        }

        let expected_len = if shape.contains(&0) {
            0
        } else {
            shape
                .iter()
                .try_fold(1_usize, |count, &extent| count.checked_mul(extent))?
        };
        if logical_len != expected_len {
            return None;
        }

        if logical_len == 0 {
            return Some(Self {
                axes: shape
                    .iter()
                    .zip(strides)
                    .map(|(&extent, &stride)| OffsetAxis {
                        extent,
                        stride,
                        position: 0,
                        rewind: 0,
                    })
                    .collect(),
                next_offset: base_offset,
                remaining: 0,
            });
        }

        let mut maximum_offset = base_offset;
        let mut axes = Vec::with_capacity(shape.len());
        for (&extent, &stride) in shape.iter().zip(strides) {
            let rewind = (extent - 1).checked_mul(stride)?;
            maximum_offset = maximum_offset.checked_add(rewind)?;
            axes.push(OffsetAxis {
                extent,
                stride,
                position: 0,
                rewind,
            });
        }
        if maximum_offset >= backing_len {
            return None;
        }

        Some(Self {
            axes,
            next_offset: base_offset,
            remaining: logical_len,
        })
    }

    fn advance(&mut self) {
        for axis in self.axes.iter_mut().rev() {
            let next_position = axis.position + 1;
            if next_position < axis.extent {
                axis.position = next_position;
                self.next_offset = self
                    .next_offset
                    .checked_add(axis.stride)
                    .expect("a checked traversal cannot overflow while advancing");
                return;
            }

            axis.position = 0;
            self.next_offset = self
                .next_offset
                .checked_sub(axis.rewind)
                .expect("a checked traversal cannot underflow while carrying");
        }

        unreachable!("a checked nonempty traversal advances within its logical shape");
    }
}

impl Iterator for StridedOffsets {
    type Item = usize;

    fn next(&mut self) -> Option<Self::Item> {
        if self.remaining == 0 {
            return None;
        }

        let offset = self.next_offset;
        self.remaining -= 1;
        if self.remaining != 0 {
            self.advance();
        }
        Some(offset)
    }

    fn size_hint(&self) -> (usize, Option<usize>) {
        (self.remaining, Some(self.remaining))
    }
}

impl ExactSizeIterator for StridedOffsets {}
impl FusedIterator for StridedOffsets {}

impl<'a> TensorView<'a> {
    /// Traverses this already validated view in logical row-major order.
    pub(crate) fn logical_offsets(&self) -> StridedOffsets {
        self.projected_offsets(&self.shape, &self.strides, self.len)
            .expect("a TensorView retains checked traversal metadata")
    }

    /// Checks effective strides for another logical traversal of this source.
    pub(crate) fn projected_offsets(
        &self,
        iteration_shape: &[usize],
        effective_strides: &[usize],
        logical_len: usize,
    ) -> Option<StridedOffsets> {
        StridedOffsets::checked(
            iteration_shape,
            effective_strides,
            self.base_offset,
            logical_len,
            self.source.len(),
        )
    }

    /// Copies one source scalar selected by an offset from a checked plan.
    pub(crate) fn value_at_storage_offset(&self, offset: usize) -> f64 {
        self.source.as_slice()[offset]
    }
}

The frozen inner-axis slice 1..3 begins at base offset 1. It keeps source strides [3,1], so its logical rows read offsets [1,2] and [4,5]. Because offset 3 is skipped, this [2,2] view is not row-major contiguous. The materializer follows those logical offsets and builds fresh storage [11,12,21,22] with strides [2,1]. Reusing the cursor does not remove this copy: materialize still allocates new storage and reads every logical value.

    /// Selects a half-open, unit-step range on one axis without copying values.
    pub fn slice(&self, axis: usize, range: Range<usize>) -> Result<Self, TensorViewError> {
        self.check_axis(axis)?;
        if range.start > range.end {
            return Err(TensorViewError::SliceStartAfterEnd {
                axis,
                start: range.start,
                end: range.end,
            });
        }

        let dimension = self.shape[axis];
        if range.end > dimension {
            return Err(TensorViewError::SliceEndOutOfBounds {
                axis,
                end: range.end,
                dimension,
            });
        }

        let start_offset = range
            .start
            .checked_mul(self.strides[axis])
            .ok_or(TensorViewError::Tensor(TensorError::ShapeOverflow))?;
        let base_offset = self
            .base_offset
            .checked_add(start_offset)
            .ok_or(TensorViewError::Tensor(TensorError::ShapeOverflow))?;
        let mut shape = self.shape.clone();
        shape[axis] = range.end - range.start;
        let (_, len) = checked_row_major_layout(&shape)?;

        Ok(Self {
            source: self.source,
            shape,
            strides: self.strides.clone(),
            base_offset,
            len,
        })
    }

    /// Copies logical row-major values into a new owned, contiguous tensor.
    pub fn materialize(&self) -> Result<Tensor, TensorViewError> {
        let mut values = Vec::with_capacity(self.len);
        for storage_offset in self.logical_offsets() {
            values.push(self.value_at_storage_offset(storage_offset));
        }
        Tensor::from_vec(self.shape.clone(), values).map_err(Into::into)
    }

The runnable learner example applies those same public methods to every success and error case, making each result traceable to one checked invariant.

    let tensor = frozen_tensor_fixture()?;
    let copied = copying_transpose(&tensor)?;
    let view = tensor.view();
    let reshaped = view.reshape(&[3, 2])?;
    let transposed = view.transpose(0, 1)?;
    let slice = view.slice(1, 1..3)?;
    let materialized = slice.materialize()?;

    let reshape_error = view.reshape(&[4, 2]).unwrap_err();
    let contiguity_error = transposed.reshape(&[2, 3]).unwrap_err();
    let slice_error = view.slice(1, 1..4).unwrap_err();

    let scalar = Tensor::from_vec(vec![], vec![7.0])?;
    let empty = Tensor::from_vec(vec![2, 0, 3], vec![])?;

See shared storage and copied storage as different states

The diagram aligns five trace records in one comparison table. The base, reshape, transpose, and slice rows all name storage owner base; a hollow diamond and the visible owner label make that shared relationship clear without color. The materialized row names its new owner and uses a filled diamond. Explicit Yes/No labels state contiguity, while each rejected request uses a multiplication sign and names the violated condition.

Reshape and transpose occupy adjacent rows because equal shape [3,2] is the tempting misconception. Compare their stride and offset columns before their value columns. The materialized row contains offsets [0,1,2,3] in its new storage; the provenance table maps source offsets [1,2,4,5] to those new positions, making the copy boundary observable.

The comparison makes each invariant explicit. Shared rows name the original owner and their logical storage offsets; the copied row names its new owner and new offsets; the provenance table connects the two. Rejected rows name the condition they violate. Read the view rows first, then the copy provenance and rejection tables.

Predict metadata and ownership before running Rust

Use the frozen owner unless a question supplies another shape.

1. Reshape [2,3] to [3,2]. Write shape, strides, logical offsets, and values.
2. Transpose axes 0 and 1. Write the same four records. Why is this not the reshape result?
3. Start with shape [2,3,4] and strides [12,4,1]. Predict shape and strides after permute([2,0,1]).
4. For slice(1, 1..3), write the new base offset, shape, strides, offsets, and values. Is it row-major contiguous?
5. Can that slice reshape directly to [4]? Give the safe two-step correction.
6. Write the materialized slice’s owner, shape, strides, offsets, and values. Why can materialize reuse one private offset cursor while public get checks every coordinate supplied by its caller?
7. Explain why changing tensor.as_mut_slice()[0] while a subsequently used borrowed view exists is rejected by Rust.

Misconception check: a transpose does not rearrange the owner’s buffer. It rearranges axis interpretation. Materialization is the operation that creates a new buffer in the view’s logical order.

Carry explicit axes into broadcasting and reductions

The course now has two complementary tensor types. Tensor owns checked, contiguous values. TensorView safely borrows those values with its own shape, strides, and base offset. Reshape, transpose, permutation, and slicing can prepare data for later numerical operations without hiding allocation.

This distinction matters throughout the decoder. Batched token states, attention heads, and matrix operands will repeatedly change axis interpretation. A checked view can express those transformations without copying scalar values; explicit materialization makes each allocation a deliberate operation. After a numerical operation validates its complete layout, the private offset cursor can traverse that layout without rebuilding a coordinate vector for every scalar. Public coordinate lookup remains checked because each call can receive a new invalid coordinate.

Chapter 10 will use the visible axes to align shapes for broadcasting and will name which axes reductions collapse. It will not infer semantic alignment merely because two views happen to touch the same storage offsets.