26 · Content revision 2

Create query, key, and value views

Learn how Transformer self-attention creates query, key, and value tensors from one hidden-state sequence through three independent bias-free projections.

Predict three outputs from one sequence

Start with one batch containing two token states:

X=\begin{bmatrix}1&2&-1\\0&1&2\end{bmatrix},\qquad X\in\mathbb{R}^{1\times2\times3}.

Use three distinct weights:

W_Q=\begin{bmatrix}1&0\\0&1\\1&-1\end{bmatrix}.

W_K=\begin{bmatrix}0&1\\1&0\\-1&1\end{bmatrix}.

W_V=\begin{bmatrix}1&1\\1&-1\\0&2\end{bmatrix}.

Before running the example, multiply the first token by each weight. The expected vectors are $q_0=[0,3]$ , $k_0=[3,0]$ , and $v_0=[3,-3]$ . The second token should produce $q_1=[2,-1]$ , $k_1=[-1,2]$ , and $v_1=[1,3]$ .

The fixture produces three different answers because its three weights differ. Independent parameter sets do not guarantee different numbers for every possible input, but they let each role learn a different representation. Batch and token coordinates remain aligned.

Project the final feature axis three ways

The complete forward rule is

Q=XW_Q,\quad K=XW_K,\quad V=XW_V

Its shape contract is

X\in\mathbb{R}^{B\times T\times d_{model}}.

W_Q,W_K,W_V\in\mathbb{R}^{d_{model}\times d_{head}}.

Q,K,V\in\mathbb{R}^{B\times T\times d_{head}}.

Every branch applies the same last-axis matrix operation independently at each batch item and token position. It does not compare tokens. In particular, these three projections alone produce no similarity scores, probabilities, causal mask, or weighted value mixture.

For the frozen fixture, the complete result is

Q=\begin{bmatrix}0&3\\2&-1\end{bmatrix},\quad K=\begin{bmatrix}3&0\\-1&2\end{bmatrix},\quad V=\begin{bmatrix}3&-3\\1&3\end{bmatrix}.

Keep roles, axes, and dimensions separate

$X$ is the hidden-state tensor entering self-attention.
$B$ is the batch size and $T$ is the number of token positions.
$d_{model}$ is the input feature width.
$d_{head}$ is the output width for this one-head chapter.
$W_Q$ , $W_K$ , and $W_V$ are independent learned weights.
$Q$ is the query view: it will ask what each position should retrieve.
$K$ is the key view: it will describe how each position can be matched.
$V$ is the value view: it carries the content that a later mixture can retrieve.

The names do not perform attention by themselves. This chapter also imposes no multi-head divisibility constraint. Chapter 30 introduces a head count $h$ and the usual relation $d_{model}=h\,d_{head}$ when it splits and merges multiple heads.

From learned alignment to self-attention projections

This history follows neural attention on the road to modern LLMs, not the history of any programming language.

Basic recurrent encoder-decoder models compressed the source into one fixed vector. Bahdanau, Cho, and Bengio, Neural Machine Translation by Jointly Learning to Align and Translate replace that bottleneck with differentiable alignment. At each target step, Bahdanau, Cho, and Bengio score every encoder annotation with the previous decoder state and use the resulting weights to form a context vector. In their notation, $s_{i-1}$ is the previous decoder state and $h_j$ is one encoder annotation entering the learned compatibility function. The query-side state and annotation-side content still come from two different parts of the encoder-decoder model.

It is useful to call the decoder state query-like and an annotation key/value-like, but this is only a retrospective bridge. Bahdanau and colleagues do not use this chapter’s query, key, and value terminology or its three-matrix layout.

Vaswani et al., Attention Is All You Need describe attention as mapping a query and key-value pairs to an output. Vaswani et al. use separate learned linear projections for queries, keys, and values and define self-attention over one sequence. Self-attention replaces two-source alignment with one previous-layer sequence feeding all three roles before scores are computed.

This separates the representations used for matching from the representation whose content will be mixed. The decoder built in this course uses the same sequence-to-three-projections pattern before causal attention. Its bias-free weights and exact dimensions are implementation choices, not claims about every Transformer.

The following comparison isolates the change in where attention inputs come from:

Contrast two attention source streams with one sequence projected three ways rust/demos/ch26-qkv-projections/src/lib.rs#historical-attention-source-contrast

fn historical_source_contrast() -> HistoryEvidence {
    HistoryEvidence {
        earlier_left: "decoder-state",
        earlier_right: "encoder-annotations",
        transformer_source: "one-sequence",
        mapping: "retrospective",
    }
}

Compose three existing differentiable linear layers

QkvProjections owns three bias-free Linear layers. Construction initializes query, key, and value weights in that stable order using a temporary copy of the seeded generator. It updates the original generator only after all three branches are valid. Manual construction checks each branch, shared dimensions, and unique names before returning the layer:

Project one rank-three hidden sequence into independent query, key, and value tensors rust/crates/llm-from-scratch/src/attention/qkv.rs#qkv-layer

/// The three projected views of the same batch and token positions.
#[derive(Clone, Debug)]
pub struct QkvForward {
    query: TensorValue,
    key: TensorValue,
    value: TensorValue,
}

impl QkvForward {
    pub fn query(&self) -> &TensorValue {
        &self.query
    }

    pub fn key(&self) -> &TensorValue {
        &self.key
    }

    pub fn value(&self) -> &TensorValue {
        &self.value
    }

    pub fn into_parts(self) -> (TensorValue, TensorValue, TensorValue) {
        (self.query, self.key, self.value)
    }
}

/// Three independent `[model_width, head_width]` linear maps with no biases.
#[derive(Clone, Debug)]
pub struct QkvProjections {
    query: Linear,
    key: Linear,
    value: Linear,
    parameters: NamedParameters,
    model_width: usize,
    head_width: usize,
}

impl QkvProjections {
    /// Initializes Q, K, and V in that order without partially advancing `rng`.
    pub fn new(
        parameter_prefix: impl Into<String>,
        model_width: usize,
        head_width: usize,
        rng: &mut SplitMix64,
    ) -> Result<Self, QkvError> {
        let parameter_prefix = parameter_prefix.into();
        let mut trial = rng.clone();
        let query = Linear::new(
            format!("{parameter_prefix}.query"),
            model_width,
            head_width,
            false,
            &mut trial,
        )
        .map_err(projection_error(QkvProjection::Query))?;
        let key = Linear::new(
            format!("{parameter_prefix}.key"),
            model_width,
            head_width,
            false,
            &mut trial,
        )
        .map_err(projection_error(QkvProjection::Key))?;
        let value = Linear::new(
            format!("{parameter_prefix}.value"),
            model_width,
            head_width,
            false,
            &mut trial,
        )
        .map_err(projection_error(QkvProjection::Value))?;
        let projections = Self::from_layers(query, key, value)?;
        *rng = trial;
        Ok(projections)
    }

    /// Gives Q/K/V semantics to three existing matrix parameters.
    pub fn from_weights(
        query_weight: NamedParameter,
        key_weight: NamedParameter,
        value_weight: NamedParameter,
    ) -> Result<Self, QkvError> {
        let query = Linear::from_parameters(query_weight, None)
            .map_err(projection_error(QkvProjection::Query))?;
        let key = Linear::from_parameters(key_weight, None)
            .map_err(projection_error(QkvProjection::Key))?;
        let value = Linear::from_parameters(value_weight, None)
            .map_err(projection_error(QkvProjection::Value))?;
        Self::from_layers(query, key, value)
    }

    fn from_layers(query: Linear, key: Linear, value: Linear) -> Result<Self, QkvError> {
        let input_widths = (query.input_width(), key.input_width(), value.input_width());
        if input_widths.0 != input_widths.1 || input_widths.0 != input_widths.2 {
            return Err(QkvError::BranchInputWidthMismatch {
                query: input_widths.0,
                key: input_widths.1,
                value: input_widths.2,
            });
        }

        let output_widths = (
            query.output_width(),
            key.output_width(),
            value.output_width(),
        );
        if output_widths.0 != output_widths.1 || output_widths.0 != output_widths.2 {
            return Err(QkvError::BranchOutputWidthMismatch {
                query: output_widths.0,
                key: output_widths.1,
                value: output_widths.2,
            });
        }

        let parameters = NamedParameters::try_new(vec![
            query.weight().clone(),
            key.weight().clone(),
            value.weight().clone(),
        ])?;
        Ok(Self {
            query,
            key,
            value,
            parameters,
            model_width: input_widths.0,
            head_width: output_widths.0,
        })
    }

    /// Projects exactly `[batch, tokens, model_width]` into three head-width views.
    pub fn forward(&self, input: &TensorValue) -> Result<QkvForward, QkvError> {
        let shape = input.shape();
        if shape.len() != 3 {
            return Err(QkvError::InputRank { rank: shape.len() });
        }
        if shape[2] != self.model_width {
            return Err(QkvError::InputWidthMismatch {
                expected: self.model_width,
                actual: shape[2],
            });
        }

        let query = self
            .query
            .forward(input)
            .map_err(projection_error(QkvProjection::Query))?;
        let key = self
            .key
            .forward(input)
            .map_err(projection_error(QkvProjection::Key))?;
        let value = self
            .value
            .forward(input)
            .map_err(projection_error(QkvProjection::Value))?;
        Ok(QkvForward { query, key, value })
    }

    pub fn query(&self) -> &Linear {
        &self.query
    }

    pub fn key(&self) -> &Linear {
        &self.key
    }

    pub fn value(&self) -> &Linear {
        &self.value
    }

    pub fn parameters(&self) -> &[NamedParameter] {
        self.parameters.as_slice()
    }

    pub const fn model_width(&self) -> usize {
        self.model_width
    }

    pub const fn head_width(&self) -> usize {
        self.head_width
    }

    pub const fn parameter_count(&self) -> usize {
        3 * self.model_width * self.head_width
    }
}

The wrapper requires exactly rank-three input so the batch and token axes remain explicit. Typed failures preserve rank-before-width and query-before-key-before-value precedence:

Keep branch and shape failures explicit rust/crates/llm-from-scratch/src/attention/qkv.rs#qkv-errors

/// The projection branch that rejected construction or a delegated operation.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum QkvProjection {
    Query,
    Key,
    Value,
}

impl fmt::Display for QkvProjection {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        formatter.write_str(match self {
            Self::Query => "query",
            Self::Key => "key",
            Self::Value => "value",
        })
    }
}

/// A rejected Q/K/V parameter set or hidden-state input.
#[derive(Clone, Debug, PartialEq)]
pub enum QkvError {
    Projection {
        projection: QkvProjection,
        source: LinearError,
    },
    InputRank {
        rank: usize,
    },
    InputWidthMismatch {
        expected: usize,
        actual: usize,
    },
    BranchInputWidthMismatch {
        query: usize,
        key: usize,
        value: usize,
    },
    BranchOutputWidthMismatch {
        query: usize,
        key: usize,
        value: usize,
    },
    Initialization(InitializationError),
}

impl fmt::Display for QkvError {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::Projection { projection, source } => {
                write!(formatter, "{projection} projection: {source}")
            }
            Self::InputRank { rank } => write!(
                formatter,
                "Q/K/V input must have rank three [batch, tokens, model_width], got rank {rank}"
            ),
            Self::InputWidthMismatch { expected, actual } => write!(
                formatter,
                "Q/K/V input final width must equal model width {expected}, got {actual}"
            ),
            Self::BranchInputWidthMismatch { query, key, value } => write!(
                formatter,
                "Q/K/V model widths must match, got query {query}, key {key}, value {value}"
            ),
            Self::BranchOutputWidthMismatch { query, key, value } => write!(
                formatter,
                "Q/K/V head widths must match, got query {query}, key {key}, value {value}"
            ),
            Self::Initialization(source) => source.fmt(formatter),
        }
    }
}

impl Error for QkvError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        match self {
            Self::Projection { source, .. } => Some(source),
            Self::Initialization(source) => Some(source),
            _ => None,
        }
    }
}

impl From<InitializationError> for QkvError {
    fn from(error: InitializationError) -> Self {
        Self::Initialization(error)
    }
}

fn projection_error(projection: QkvProjection) -> impl FnOnce(LinearError) -> QkvError {
    move |source| QkvError::Projection { projection, source }
}

The reverse fixture forms one scalar objective,

L=\langle Q,U_Q\rangle+\langle K,U_K\rangle+\langle V,U_V\rangle.

If bars denote reverse-mode gradients, all three paths accumulate into the gradient of the shared input:

\bar X=\bar QW_Q^{\mathsf T}+\bar KW_K^{\mathsf T}+\bar VW_V^{\mathsf T}.

For the weights, flatten the batch and token axes into one row axis, denoted by the subscript $(BT)$ . Each branch then keeps its own gradient:

\bar W_Q=X_{(BT)}^{\mathsf T}\bar Q_{(BT)},\quad \bar W_K=X_{(BT)}^{\mathsf T}\bar K_{(BT)},\quad \bar W_V=X_{(BT)}^{\mathsf T}\bar V_{(BT)}.

With the fixture’s upstream gradients, the shared-input result is

\bar X=\begin{bmatrix}3&1.5&1.5\\-1.5&3.5&-5\end{bmatrix},

while each branch keeps its own weight gradient. Central differences check all six coordinates of $X$ , $W_Q$ , $W_K$ , and $W_V$ with step $10^{-6}$ and tolerance $2\times10^{-6}$ . Empty batch and token axes remain connected to the gradient tape, and two runs replay by exact floating-point bit pattern:

Build exact forward, reverse, shape, error, initialization, and gradient evidence rust/demos/ch26-qkv-projections/src/lib.rs#qkv-fixture

pub fn learner_evidence() -> Result<LearnerEvidence, FixtureError> {
    let primary = primary_once()?;
    let replay = primary_once()?;
    let (query_changed, key_unchanged, value_unchanged) = independence_evidence()?;
    let (input_checks, query_weight_checks, key_weight_checks, value_weight_checks, passed) =
        gradient_evidence(&primary)?;
    Ok(LearnerEvidence {
        replay_bitwise: same_primary_bits(&primary, &replay),
        primary,
        shapes: shape_evidence()?,
        errors: error_evidence()?,
        initialization: initialization_evidence()?,
        history: historical_source_contrast(),
        query_changed,
        key_unchanged,
        value_unchanged,
        input_checks,
        query_weight_checks,
        key_weight_checks,
        value_weight_checks,
        gradcheck_passed: passed,
    })
}

Run the complete query, key, and value projection example rust/demos/ch26-qkv-projections/src/main.rs

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let evidence = ch26_qkv_projections::learner_evidence()?;
    print!("{}", ch26_qkv_projections::render_report(&evidence));
    Ok(())
}

Run cargo run --quiet --locked -p ch26-qkv-projections to inspect the same forward values, gradients, shape probes, and rejected inputs from the executable example.

Inspect how one sequence becomes three learned representations

The diagram brings the shared input, three projection weights, exact outputs, combined reverse path, branch-local gradients, empty shapes, rejected inputs, and historical source comparison into one view:

Split one hidden sequence into three learned views

Trace one hidden-state sequence through query, key, and value projections, then compare their shapes, gradients, independence, historical sources, and rejected inputs.

Shared input border
Solid query border
Dashed key border
Double value border

Preserve positions while changing feature space

Shared hidden-state input

X=[1.000000,2.000000,-1.000000,0.000000,1.000000,2.000000]

Shape: $\operatorname{shape}(X)=[1,2,3]$
Bias policy: bias=false

Solid query border

Query projection

Query: what should this position retrieve?

$Q=XW_Q$

Stable parameter: decoder.block.0.attention.query.weight
Projection weight: $\operatorname{shape}(W_Q)=[3,2]$
Projected output: $\operatorname{shape}(Q)=[1,2,2]$

$W_Q$
	$0$	$1$
$0$	$1.000000$	$0.000000$
$1$	$0.000000$	$1.000000$
$2$	$1.000000$	$-1.000000$

Projected output
	$0$	$1$
$t_0$	$0.000000$	$3.000000$
$t_1$	$2.000000$	$-1.000000$

Dashed key border

Key projection

Key: how can this position be matched?

$K=XW_K$

Stable parameter: decoder.block.0.attention.key.weight
Projection weight: $\operatorname{shape}(W_K)=[3,2]$
Projected output: $\operatorname{shape}(K)=[1,2,2]$

$W_K$
	$0$	$1$
$0$	$0.000000$	$1.000000$
$1$	$1.000000$	$0.000000$
$2$	$-1.000000$	$1.000000$

Projected output
	$0$	$1$
$t_0$	$3.000000$	$0.000000$
$t_1$	$-1.000000$	$2.000000$

Double value border

Value projection

Value: what content can this position contribute?

$V=XW_V$

Stable parameter: decoder.block.0.attention.value.weight
Projection weight: $\operatorname{shape}(W_V)=[3,2]$
Projected output: $\operatorname{shape}(V)=[1,2,2]$

$W_V$
	$0$	$1$
$0$	$1.000000$	$1.000000$
$1$	$1.000000$	$-1.000000$
$2$	$0.000000$	$2.000000$

Projected output
	$0$	$1$
$t_0$	$3.000000$	$-3.000000$
$t_1$	$1.000000$	$3.000000$

The three branches share coordinates, not weights or feature values.

Change from two attention sources to one

Query-like and key/value-like are a limited retrospective analogy for the earlier two-stream mechanism.
	Previous decoder state	Encoder annotations	One previous-layer sequence
Additive encoder-decoder attention	`decoder-state`	`encoder-annotations`	—
Transformer self-attention	—	—	`one-sequence`

Query-like and key/value-like are a limited retrospective analogy for the earlier two-stream mechanism.

Check independence, gradients, and boundaries

Combined input gradient

$\bar X=\begin{bmatrix}3.000000&1.500000&1.500000\\-1.500000&3.500000&-5.000000\end{bmatrix}$

$\operatorname{shape}(\bar X)=[1,2,3]$

Branch-local weight gradients

Query: what should this position retrieve?	$\bar W_Q=[1.000000,0.000000,1.000000,2.000000,-3.000000,4.000000]$
Key: how can this position be matched?	$\bar W_K=[0.500000,-1.000000,2.000000,-2.000000,1.500000,1.000000]$
Value: what content can this position contribute?	$\bar W_V=[2.000000,1.000000,4.000000,1.500000,-2.000000,-2.000000]$

Change only the query weight

Query: what should this position retrieve?: Changed
Key: how can this position be matched?: Unchanged
Value: what content can this position contribute?: Unchanged

Empty batch axis

$[0,2,3]\to [0,2,2],[0,2,2],[0,2,2]$

Empty token axis

$[2,0,3]\to [2,0,2],[2,0,2],[2,0,2]$

Rejected boundaries

Rejected rank-two The input must keep explicit batch, token, and feature axes.
Rejected input-width The final input axis must match the model width.
Rejected branch-mismatch All three projection weights must use the same model width.

Numerical checks

$n_X=6, n_Q=6, n_K=6, n_V=6$

$\tau=0.000002$

gradcheck=true replay=bitwise names=unique initialization=transactional

The fixture checks every input and weight coordinate and confirms repeatable, independent projections.

Read the three branches from the same input coordinate. Their solid, dashed, and double borders distinguish query, key, and value without relying on color. The diagram stops before comparing queries with keys: it shows the representations that attention will consume, not an attention decision.

Predict before reading the evidence

Predict all three output shapes for input $[4,7,3]$ when $d_{head}=2$ .
Compute $q_0$ , $k_0$ , and $v_0$ for the first frozen token.
Decide whether changing only $W_Q$ can change $K$ or $V$ .
Count the trainable scalars in three bias-free $[3,2]$ weights.
Predict the three outputs’ shapes for inputs $[0,2,3]$ and $[2,0,3]$ .
Explain why rank-two $[T,d_{model}]$ input is rejected by this wrapper.
Decide whether $d_{model}$ must be divisible by $d_{head}$ here.
Identify which historical mechanism uses two source streams and which uses one.
State which computation is still missing before these tensors form an attention output.

Check the predictions

Each output has shape $[4,7,2]$ .
The vectors are $[0,3]$ , $[3,0]$ , and $[3,-3]$ .
No. Independent weights make the unchanged key and value outputs replay bitwise.
The count is $3\times3\times2=18$ .
The outputs have shapes $[0,2,2]$ and $[2,0,2]$ for every branch.
The API keeps both batch and token axes explicit, so it requires rank three.
No. Divisibility belongs to later multi-head splitting.
Additive encoder-decoder attention uses decoder and encoder streams; self-attention projects one sequence three ways.
Chapter 27 must compute query-key scores, probabilities, and a weighted value mixture.

Compare queries with keys and mix values next

The cumulative decoder now accepts a normalized hidden sequence with shape $[B,T,d_{model}]$ and emits separate $Q$ , $K$ , and $V$ tensors with shape $[B,T,d_{head}]$ . Chapter 27 will turn those prepared feature views into one unmasked attention head by computing scores, probabilities, and a value mixture.