29 · Content revision 3

Turn query and key pairs with RoPE

Learn how rotary position embeddings turn query and key feature pairs by absolute position so relative offsets appear in attention dot products, with a tested Rust implementation.

Predict one pair before looking at the table

Use one two-coordinate query/key pair and the first frequency:

q=[1,0],\qquad k=[0,1],\qquad \theta_0=1.

At equal positions $m=n=0$ , neither vector rotates, so

q^\top k=0.

Now put the query at $m=1$ while the key stays at $n=0$ :

R(1)q\approx[0.540302,0.841471],\qquad R(0)k=[0,1].

Their dot product is approximately $0.841471$ . Shift both absolute positions by $2$ : $(m,n)=(3,2)$ still gives $n-m=-1$ , so the dot product for these unchanged vectors remains approximately $0.841471$ . Both rotated coordinates change, but their relative rotation does not.

The executable example expands this idea to $d=4$ . It repeats $[1,0,1,0]$ at positions $0$ , $1$ , and $2$ , uses $b=100$ , and therefore has frequencies

[\theta_0,\theta_1]=[1,0.1].

Before running it, predict which pair turns faster, what happens at position $0$ , and which cells should match after every query and key position is shifted by $3$ while their contents stay fixed.

Rotate adjacent coordinates by absolute position

For each adjacent feature pair,

\left(\operatorname{RoPE}(x_m)\right)_{2k:2k+2}=R(m\theta_k)(x_m)_{2k:2k+2}

The rotation matrix and frequency schedule are

R(\phi)= \begin{bmatrix} \cos\phi&-\sin\phi\\ \sin\phi&\cos\phi \end{bmatrix}, \qquad \theta_k=b^{-2k/d}.

This implementation pairs coordinates $2k$ and $2k+1$ . Other implementations may arrange the pairs differently. Here $d$ must be nonzero and even so every coordinate has a partner.

Position $m=0$ gives a zero angle for every pair, and each rotation is orthogonal:

R(0)=I, \qquad R(\phi)^\top R(\phi)=I, \qquad \lVert R(\phi)x\rVert_2=\lVert x\rVert_2.

When a query at $m$ meets a key at $n$ ,

\bigl(R(m\theta_k)q_m^{(k)}\bigr)^\top \bigl(R(n\theta_k)k_n^{(k)}\bigr) =(q_m^{(k)})^\top R((n-m)\theta_k)k_n^{(k)}.

This follows from

R(\alpha)^\top R(\beta)=R(\beta-\alpha).

The rotations receive absolute positions. The signed difference $n-m$ appears only when the two rotations combine in the dot product. The score still depends on the query/key contents and every $\theta_k$ ; it is not a function of position difference alone.

Let $L$ be a scalar loss, $\bar{y}=\partial L/\partial y$ the output adjoint, and $\bar{x}=\partial L/\partial x$ the input adjoint. Reverse mode applies the transposed rotation:

\begin{bmatrix}\bar{x}_{2k}\\\bar{x}_{2k+1}\end{bmatrix} =R(m\theta_k)^\top \begin{bmatrix}\bar{y}_{2k}\\\bar{y}_{2k+1}\end{bmatrix}.

Keep position, pair, and visibility roles distinct

$q_m^{(k)}$ and $k_n^{(k)}$ are the two-coordinate query and key pairs at absolute positions $m$ and $n$ before rotation.
$k$ selects one adjacent coordinate pair and its frequency $\theta_k$ .
$\phi$ is any rotation angle, and $I$ is the identity matrix.
$n-m$ is the signed position of the key relative to the query, not an unsigned distance.
A bar denotes a derivative of $L$ : $\bar{y}$ arrives from later operations, and $\bar{x}$ is sent to earlier ones.

Without a position signal and before masking, content-only self-attention is equivariant to a consistent permutation of its rows: permuting queries, keys, and values together merely permutes the outputs. RoPE adds relative-position geometry to query-key scores. It does not decide visibility. Chapter 28’s causal mask still blocks future keys, and values $V$ are not rotated here.

From recurrent order to rotary score geometry

Recurrent neural language models carried order through a state updated one token at a time; unmasked self-attention without a position signal is instead equivariant to a consistent permutation of its content rows.

Vaswani et al. describe the original Transformer. Vaswani et al. remove recurrence and convolution, add positional encodings to input embeddings, and compare fixed sinusoidal and learned position representations. Their sinusoidal choice was motivated by a hypothesis about learning fixed offsets, not a guarantee of extrapolation to every longer context.

The additive approach changes an embedding before it becomes a query, key, or value. This small Rust function shows that operation directly:

Add a sinusoidal position vector to one embedding rust/demos/ch29-rope/src/lib.rs#historical-position-contrast

/// Add the original Transformer's sinusoidal position vector to one embedding.
/// RoPE instead leaves the embedding unchanged and rotates query/key pairs later.
pub fn add_sinusoidal_position(mut embedding: [f64; 4], position: usize) -> [f64; 4] {
    let position = position as f64;
    let fast_angle = position;
    let slow_angle = position / 100.0;
    embedding[0] += fast_angle.sin();
    embedding[1] += fast_angle.cos();
    embedding[2] += slow_angle.sin();
    embedding[3] += slow_angle.cos();
    embedding
}

Su et al. introduce RoFormer. Su et al. encode absolute position through rotations of query and key subspaces and derive a self-attention inner product whose positional term uses their position difference. Unlike the original Transformer’s additive input encoding, RoFormer’s query-key rotations introduce an inner-product term that depends on the signed position difference.

Touvron et al. describe the original LLaMA. Touvron et al. document that the original LLaMA removes absolute positional embeddings and applies RoPE at each Transformer layer. In the original LLaMA, RoPE shapes query-key score geometry at each Transformer layer while the causal mask separately controls visibility. This example does not make RoPE universal, guarantee length extrapolation, or prove that an entire decoder is invariant to a shifted input.

Precompute once, rotate each query and key row

RotaryEmbedding::new validates a nonzero even width, a nonzero position capacity, and a positive finite base. The complete table specification also checks that the number of position-pair cells fits in usize, that each array length is a representable Vec<f64> capacity, and that every inverse frequency is finite and nonnegative. For one pair, multiplying that frequency by larger nonnegative positions cannot produce a smaller angle. A finite angle also has a finite sine and cosine. The storage-free check can therefore prove that every table cell is finite by testing the last position for each pair. If that last angle is not finite, binary search finds the pair’s first failing position; the smallest resulting position-pair index preserves row-major error order.

This bounded check grows with the number of coordinate pairs and, only for a failing pair, the logarithm of the position capacity—not with the number of table cells. It allocates no table storage and does not test whether the allocator can satisfy an otherwise representable request; only construction can report that failure. Construction then reserves the three arrays, visits every cell in row-major order, and retains one inverse frequency per pair and one sine/cosine row per absolute position. A decoder can thus validate the logical RoPE table implied by an existing attention layout without building a temporary table, while the actual attention component still owns the tables it constructs.

Prove every rotary table value is finite, then materialize the table only during RoPE construction rust/crates/llm-from-scratch/src/attention/rope.rs#rope-tables

    /// Precomputes one sine/cosine row per absolute position.
    pub fn new(feature_width: usize, max_positions: usize, base: f64) -> Result<Self, RopeError> {
        let layout = Self::validate_table_layout(feature_width, max_positions, base)?;
        let mut frequencies = reserved_values(layout.pairs)?;
        Self::visit_inverse_frequencies(feature_width, base, |_, frequency| {
            frequencies.push(frequency);
        })?;

        let mut cosines = reserved_values(layout.table_elements)?;
        let mut sines = reserved_values(layout.table_elements)?;
        Self::visit_table_values(
            max_positions,
            layout.pairs,
            |pair| frequencies[pair],
            |_, _, sine, cosine| {
                cosines.push(cosine);
                sines.push(sine);
            },
        )?;

        Ok(Self {
            feature_width,
            max_positions,
            base,
            inverse_frequencies: Tensor::from_vec(vec![layout.pairs], frequencies)?,
            cosines: Tensor::from_vec(vec![max_positions, layout.pairs], cosines)?,
            sines: Tensor::from_vec(vec![max_positions, layout.pairs], sines)?,
        })
    }

    /// Proves the deterministic table is finite without allocating its storage.
    pub(crate) fn validate_table_specification(
        feature_width: usize,
        max_positions: usize,
        base: f64,
    ) -> Result<(), RopeError> {
        let layout = Self::validate_table_layout(feature_width, max_positions, base)?;
        Self::validate_value_capacity(layout.pairs)?;
        Self::visit_inverse_frequencies(feature_width, base, |_, _| {})?;
        Self::validate_value_capacity(layout.table_elements)?;
        Self::validate_table_values(max_positions, layout.pairs, |pair| {
            Self::inverse_frequency(feature_width, base, pair)
        })
    }

    fn validate_table_layout(
        feature_width: usize,
        max_positions: usize,
        base: f64,
    ) -> Result<RotaryTableLayout, RopeError> {
        if feature_width == 0 {
            return Err(RopeError::ZeroFeatureWidth);
        }
        if !feature_width.is_multiple_of(2) {
            return Err(RopeError::OddFeatureWidth {
                width: feature_width,
            });
        }
        if max_positions == 0 {
            return Err(RopeError::ZeroPositionCapacity);
        }
        if !base.is_finite() || base <= 0.0 {
            return Err(RopeError::InvalidBase { base });
        }

        let pairs = feature_width / 2;
        let table_elements =
            max_positions
                .checked_mul(pairs)
                .ok_or(RopeError::TableSizeOverflow {
                    positions: max_positions,
                    pairs,
                })?;
        Ok(RotaryTableLayout {
            pairs,
            table_elements,
        })
    }

    fn validate_value_capacity(elements: usize) -> Result<(), RopeError> {
        let maximum = (isize::MAX as usize) / std::mem::size_of::<f64>();
        if elements > maximum {
            return Err(RopeError::TableAllocationFailed { elements });
        }
        Ok(())
    }

    fn inverse_frequency(feature_width: usize, base: f64, pair: usize) -> f64 {
        let exponent = -2.0 * (pair as f64) / (feature_width as f64);
        base.powf(exponent)
    }

    fn visit_inverse_frequencies(
        feature_width: usize,
        base: f64,
        mut visit: impl FnMut(usize, f64),
    ) -> Result<(), RopeError> {
        let pairs = feature_width / 2;
        for pair in 0..pairs {
            let frequency = Self::inverse_frequency(feature_width, base, pair);
            if !frequency.is_finite() {
                return Err(RopeError::NonFiniteTableValue { position: 0, pair });
            }
            visit(pair, frequency);
        }
        Ok(())
    }

    fn visit_table_values(
        max_positions: usize,
        pairs: usize,
        mut frequency: impl FnMut(usize) -> f64,
        mut visit: impl FnMut(usize, usize, f64, f64),
    ) -> Result<(), RopeError> {
        for position in 0..max_positions {
            for pair in 0..pairs {
                let (sine, cosine) = Self::table_value(position, frequency(pair))
                    .ok_or(RopeError::NonFiniteTableValue { position, pair })?;
                visit(position, pair, sine, cosine);
            }
        }
        Ok(())
    }

    fn validate_table_values(
        max_positions: usize,
        pairs: usize,
        mut frequency: impl FnMut(usize) -> f64,
    ) -> Result<(), RopeError> {
        let last_position = max_positions - 1;
        let mut first_nonfinite = None;
        for pair in 0..pairs {
            let frequency = frequency(pair);
            debug_assert!(frequency.is_finite());
            if Self::table_value(last_position, frequency).is_some() {
                continue;
            }

            let mut last_finite = 0;
            let mut first_invalid = last_position;
            debug_assert!(Self::table_value(last_finite, frequency).is_some());
            while last_finite + 1 < first_invalid {
                let middle = last_finite + (first_invalid - last_finite) / 2;
                if Self::table_value(middle, frequency).is_some() {
                    last_finite = middle;
                } else {
                    first_invalid = middle;
                }
            }
            let candidate = (first_invalid, pair);
            first_nonfinite = Some(match first_nonfinite {
                Some(current) => std::cmp::min(current, candidate),
                None => candidate,
            });
        }
        match first_nonfinite {
            Some((position, pair)) => Err(RopeError::NonFiniteTableValue { position, pair }),
            None => Ok(()),
        }
    }

    fn table_value(position: usize, frequency: f64) -> Option<(f64, f64)> {
        let angle = (position as f64) * frequency;
        let (sine, cosine) = angle.sin_cos();
        (angle.is_finite() && sine.is_finite() && cosine.is_finite())
            .then(|| (canonical_zero(sine), canonical_zero(cosine)))
    }

RotaryEmbedding::rotate interprets the penultimate axis as tokens and the final axis as features. It accepts $[T,d]$ , $[B,T,d]$ , $[B,H,T,d]$ , and independent leading axes. For supplied offset $o$ , it checks the interval

[o,o+T).

An empty token interval may start at the table capacity; a nonempty interval may not cross it.

Validate shapes and select the absolute-position interval rust/crates/llm-from-scratch/src/attention/rope.rs#rope-rotation

    /// Rotates the final feature axis at consecutive absolute positions.
    ///
    /// The penultimate axis is the token axis. Any earlier axes are independent
    /// lanes, so rank-two, batched, and batched-head layouts share one rule.
    pub fn rotate(
        &self,
        input: &TensorValue,
        position_offset: usize,
    ) -> Result<TensorValue, RopeError> {
        let shape = input.shape();
        if shape.len() < 2 {
            return Err(RopeError::InputRank { rank: shape.len() });
        }
        let actual_width = shape[shape.len() - 1];
        if actual_width != self.feature_width {
            return Err(RopeError::FeatureWidthMismatch {
                expected: self.feature_width,
                actual: actual_width,
            });
        }
        let tokens = shape[shape.len() - 2];
        let position_end =
            position_offset
                .checked_add(tokens)
                .ok_or(RopeError::PositionOffsetOverflow {
                    offset: position_offset,
                    tokens,
                })?;
        if position_end > self.max_positions {
            return Err(RopeError::PositionRangeExceeded {
                offset: position_offset,
                tokens,
                max_positions: self.max_positions,
            });
        }

        let pairs = self.feature_width / 2;
        let start = position_offset
            .checked_mul(pairs)
            .ok_or(RopeError::TableSizeOverflow {
                positions: position_offset,
                pairs,
            })?;
        let end = position_end
            .checked_mul(pairs)
            .ok_or(RopeError::TableSizeOverflow {
                positions: position_end,
                pairs,
            })?;
        let table_shape = [tokens, pairs];
        let cosines = copy_table_slice(&self.cosines.as_slice()[start..end], &table_shape)?;
        let sines = copy_table_slice(&self.sines.as_slice()[start..end], &table_shape)?;
        input.rotary_pairs(&cosines, &sines).map_err(Into::into)
    }

The differentiable operation visits each adjacent pair once, without building a dense $d\times d$ matrix. Its inverse branch applies the transpose for the vector-Jacobian product:

Rotate adjacent pairs and reverse them with the transpose rust/crates/llm-from-scratch/src/autograd/model_ops.rs#rotary-pairs-forward

fn rotary_pairs_forward(
    input: &Tensor,
    cosines: &Tensor,
    sines: &Tensor,
    inverse: bool,
) -> Result<Tensor, TensorAutodiffError> {
    debug_assert!(input.rank() >= 2);
    debug_assert_eq!(cosines.shape(), sines.shape());
    debug_assert_eq!(cosines.rank(), 2);

    let width = input.shape()[input.rank() - 1];
    let tokens = input.shape()[input.rank() - 2];
    let pairs = width / 2;
    debug_assert_eq!(cosines.shape(), [tokens, pairs]);

    let mut output = zeros(input.shape())?;
    if input.is_empty() {
        return Ok(output);
    }

    let rows = input.len() / width;
    for row in 0..rows {
        let token = row % tokens;
        for pair in 0..pairs {
            let feature = row * width + pair * 2;
            let table = token * pairs + pair;
            let left = input.as_slice()[feature];
            let right = input.as_slice()[feature + 1];
            let cosine = cosines.as_slice()[table];
            let sine = sines.as_slice()[table];
            let (rotated_left, rotated_right) = if inverse {
                (left * cosine + right * sine, -left * sine + right * cosine)
            } else {
                (left * cosine - right * sine, left * sine + right * cosine)
            };
            output.as_mut_slice()[feature] = canonical_zero(rotated_left);
            output.as_mut_slice()[feature + 1] = canonical_zero(rotated_right);
        }
    }
    Ok(output)
}

fn canonical_zero(value: f64) -> f64 {
    if value == 0.0 { 0.0 } else { value }
}

The example calls the same operation independently for $Q$ and $K$ . Central differences check all $12$ coordinates of each input with

h=10^{-6},\qquad \varepsilon_g=4\times10^{-6}.

It also checks signs, norms, every dot-matrix cell, an equal position shift, rank-two through rank-four layouts, empty axes, invalid configurations, position-range errors, finite saved values, and deterministic replay. Run cargo run --quiet --locked -p ch29-rope to inspect the complete values. The example executable constructs and prints that deterministic report:

Run the complete RoPE example rust/demos/ch29-rope/src/main.rs

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let evidence = ch29_rope::learner_evidence()?;
    print!("{}", ch29_rope::render_report(&evidence));
    Ok(())
}

Read the pair rotations and relative diagonals

Watch absolute rotations reveal relative query-key positions

Compare two pair frequencies, their rotations, a query-key dot matrix, equal-shift evidence, reverse-mode values, valid shapes, rejected boundaries, and the path to modern decoder LLMs.

Fast pair: solid cue
Slow pair: dashed cue
Equal positions: double cue
Key lies later: solid cue
Key lies earlier: dashed cue

Follow two feature pairs through three absolute positions

The first pair advances one radian per position; the second advances one tenth of a radian.

\left(\operatorname{RoPE}(x_m)\right)_{2k:2k+2}=R(m\theta_k)(x_m)_{2k:2k+2}

The first pair advances one radian per position; the second advances one tenth of a radian.
Absolute position	Coordinate pair	Feature coordinates	Radians per position	Absolute angle	Before rotation	After rotation
$m=0$	$k=0$ Fast pair: solid cue	$[0,1]$	$\theta_0=1.000000$	$0.000000$	$[1.000000,0.000000]$	$[1.000000,0.000000]$
$m=0$	$k=1$ Slow pair: dashed cue	$[2,3]$	$\theta_1=0.100000$	$0.000000$	$[1.000000,0.000000]$	$[1.000000,0.000000]$
$m=1$	$k=0$ Fast pair: solid cue	$[0,1]$	$\theta_0=1.000000$	$1.000000$	$[1.000000,0.000000]$	$[0.540302,0.841471]$
$m=1$	$k=1$ Slow pair: dashed cue	$[2,3]$	$\theta_1=0.100000$	$0.100000$	$[1.000000,0.000000]$	$[0.995004,0.099833]$
$m=2$	$k=0$ Fast pair: solid cue	$[0,1]$	$\theta_0=1.000000$	$2.000000$	$[1.000000,0.000000]$	$[-0.416147,0.909297]$
$m=2$	$k=1$ Slow pair: dashed cue	$[2,3]$	$\theta_1=0.100000$	$0.200000$	$[1.000000,0.000000]$	$[0.980067,0.198669]$

Read signed relative positions in the query-key dot matrix

Because this example repeats the same content, cells with the same signed relative position repeat along diagonals.

\left(R(m\theta_k)q\right)^\top\left(R(n\theta_k)k\right)=q^\top R((n-m)\theta_k)k

**Rotated dot product** — Key position relative to query. Because this example repeats the same content, cells with the same signed relative position repeat along diagonals.
Query position / Key position	$n=0$	$n=1$	$n=2$
$m=0$	$2.000000$ $n-m=0$ · Equal positions: double cue	$1.535306$ $n-m=1$ · Key lies later: solid cue	$0.563920$ $n-m=2$ · Key lies later: solid cue
$m=1$	$1.535306$ $n-m=-1$ · Key lies earlier: dashed cue	$2.000000$ $n-m=0$ · Equal positions: double cue	$1.535306$ $n-m=1$ · Key lies later: solid cue
$m=2$	$0.563920$ $n-m=-2$ · Key lies earlier: dashed cue	$1.535306$ $n-m=-1$ · Key lies earlier: dashed cue	$2.000000$ $n-m=0$ · Equal positions: double cue

Dot matrix at positions zero through two

Original absolute positions: $[0,1,2]$

Dot matrix at positions zero through two
Query position / Key position	$n=0$	$n=1$	$n=2$
$m=0$	$2.000000$	$1.535306$	$0.563920$
$m=1$	$1.535306$	$2.000000$	$1.535306$
$m=2$	$0.563920$	$1.535306$	$2.000000$

Dot matrix at positions three through five

Equally shifted positions: $[3,4,5]$

Dot matrix at positions three through five
Query position / Key position	$n=3$	$n=4$	$n=5$
$m=3$	$2.000000$	$1.535306$	$0.563920$
$m=4$	$1.535306$	$2.000000$	$1.535306$
$m=5$	$0.563920$	$1.535306$	$2.000000$

Fixed-content equal-shift check

Checked

$\varepsilon=0.000000000001$

Check norms, shapes, boundaries, and reverse mode

Orthogonal rotations preserve norms and shapes; the transpose carries output adjoints back to the inputs.

Norm preservation

Before rotation: $[1.414214,1.414214,1.414214]$
After rotation: $[1.414214,1.414214,1.414214]$
Equally shifted positions: $[1.414214,1.414214,1.414214]$

Checked

Preserved tensor shapes

Batch, token, and feature axes $[2,3,4]\to[2,3,4]$ Checked
Batch, head, token, and feature axes $[2,2,3,4]\to[2,2,3,4]$ Checked
Empty leading axis $[0,3,4]\to[0,3,4]$ Checked
Empty token axis at table capacity $[2,0,4]\to[2,0,4]$ Checked

Rejected boundaries

Odd final feature width odd-feature-width Checked
Input below rank two input-rank Checked
Configured and input widths differ feature-width-mismatch Checked
Position interval exceeds the table position-range-exceeded Checked
Position interval overflows position-offset-overflow Checked
Released autodiff operand autodiff-stage Checked

Transpose rotation in reverse mode
Input tensor	Output adjoint	Input adjoint
$Q$	$\begin{bmatrix}1.000000&-0.500000&0.250000&0.750000\\-0.300000&0.800000&1.200000&-0.400000\\0.600000&0.100000&-0.700000&0.900000\end{bmatrix}$	$\begin{bmatrix}1.000000&-0.500000&0.250000&0.750000\\0.511086&0.684683&1.154072&-0.517802\\-0.158758&-0.587193&-0.507244&1.021128\end{bmatrix}$
$K$	$\begin{bmatrix}-0.200000&0.400000&0.900000&-0.600000\\0.500000&1.100000&-0.800000&0.300000\\1.000000&-0.900000&0.200000&0.700000\end{bmatrix}$	$\begin{bmatrix}-0.200000&0.400000&0.900000&-0.600000\\1.195769&0.173597&-0.766053&0.378368\\-1.234515&-0.534765&0.335082&0.646313\end{bmatrix}$

All-coordinate gradient check:

12+12

\varepsilon_g=0.000004

· Checked

Follow position information toward modern decoder LLMs

Position information moved from sequential state to explicit vectors and then into query-key score geometry.

Order carried by recurrent state

A recurrent state changed once per token, carrying order through sequential computation.
Position vectors added to embeddings

The original Transformer added explicit position vectors to token embeddings.
Query-key pair rotations

RoPE rotates query and key pairs by absolute position so their dot product contains a signed relative-position term.
RoPE in an influential decoder LLM

The original LLaMA applies RoPE at every Transformer layer.

Visibility remains a separate decision

RoPE changes score geometry. The causal mask still decides which keys a query may see.

The matrix before the equal shift is

\begin{bmatrix} 2&1.535306&0.563920\\ 1.535306&2&1.535306\\ 0.563920&1.535306&2 \end{bmatrix}.

Because this example repeats the same query and key content at every position, equal signed offsets repeat along diagonals. Moving the unchanged content from absolute positions $0,1,2$ to $3,4,5$ changes the rotated coordinates but keeps all nine dot products equal within $10^{-12}$ . This numerical comparison checks one instance of the algebraic identity; the identity itself is established by the rotation equation above.

Predict, then check the invariant

Prove that position $m=0$ is the identity for every pair.
Compute the single-pair result for $(m,n)=(1,0)$ .
Predict what changes after adding the same offset to both positions while holding query and key contents fixed.
Explain why shifting only $m$ generally changes the dot product.
Derive norm preservation from $R(\phi)^\top R(\phi)=I$ .
Decide whether $V$ or the causal mask should be rotated.
Explain why this full-width adjacent-pair implementation rejects $d=3$ .
Distinguish the absolute inputs from the signed difference in the dot product.
Explain why the local fixed-content identity does not prove shift invariance for an entire decoder.

Check the key distinctions

Position $m=0$ gives angle $0$ for every pair, so $R(0)=I$ .
At $(m,n)=(1,0)$ , the dot product is approximately $0.841471$ .
Adding $c$ to both positions keeps $(n+c)-(m+c)=n-m$ , so the dot product for unchanged contents stays fixed although both absolute rotations change.
Shifting only $m$ changes $n-m$ and therefore generally changes the result; special contents or periodic angles can still produce equality.
$R(\phi)^\top R(\phi)=I$ preserves squared norm and therefore norm.
This chapter rotates $Q$ and $K$ , not $V$ ; the causal mask independently decides which keys are visible.
Width $3$ leaves one coordinate without a partner.
Absolute indices set local angles; the signed relative position appears only after the query and key rotations combine.
A whole-model invariance claim would require checking every component and boundary and preserving the same content and context. This local identity alone is insufficient.

Misconception: RoPE receives relative indices.

Correction: it receives absolute positions. Shared rotation algebra makes a fixed query-key dot product depend on their signed relative difference. RoPE does not replace causal masking, and the score still depends on content and frequency.

Hand the position-aware axis to multiple heads

The current single-head path rotates each query and key row before Chapter 28’s causal score computation. Chapter 30 first reshapes projected queries and keys into heads, applies RoPE along each head’s final feature axis, performs causal attention independently per head, concatenates the head outputs, and applies the learned output projection $W_O$ .

Partial rotary dimensions, alternate pair layouts, key/value caches, prefill, and long-context frequency scaling remain outside this chapter.