19 · Content revision 5

Mix each token's features with one learned projection

Build a trainable linear layer in Rust, preserve leading token axes, compare affine and bias-free projections, and verify exact reverse gradients.

Predict two outputs from one shared matrix

Chapter 18 left one feature vector at each token position. Use the two vectors $[1,2]$ and $[-1,3]$ as $X$ with shape $[1,2,2]$ , then share these parameters across both positions:

W= \begin{bmatrix} 1 & 0 & -1 \\ 2 & 0.5 & 1 \end{bmatrix}, \qquad b=\begin{bmatrix}0.5 & -0.5 & 1\end{bmatrix}.

Before running the example, predict six outputs. For $[1,2]$ , the three weighted sums are $[5,1,1]$ ; adding $b$ gives $[5.5,0.5,2]$ . For $[-1,3]$ , the weighted sums are $[5,1.5,4]$ ; adding that same $b$ gives $[5.5,1,5]$ . The complete output therefore has shape $[1,2,3]$ and values $[[[5.5,0.5,2],[5.5,1,5]]]$ .

One scalar weighted response is useful as a small algebraic contrast. It equals the first output coordinate, but it is not a separate implementation of the matrix operation:

Compute one scalar weighted response for the bounded historical contrast rust/demos/ch19-linear-layers/src/lib.rs#scalar-weighted-unit

/// Computes one historical scalar weighted response for comparison.
pub fn scalar_weighted_response(input: &[f64], weights: &[f64], bias: f64) -> f64 {
    assert_eq!(input.len(), weights.len());
    input
        .iter()
        .zip(weights)
        .map(|(feature, weight)| feature * weight)
        .sum::<f64>()
        + bias
}

The exact fixture runs both the affine path and the course decoder’s bias-free policy through one shared layer:

Apply the worked affine and bias-free projections rust/demos/ch19-linear-layers/src/lib.rs#known-linear-layer

    let layer = known_linear(true);
    let input = TensorValue::parameter(tensor(&INPUT_SHAPE, &INPUT_VALUES))?;
    let output = layer.forward(&input)?;
    let bias_free = known_linear(false);
    let bias_free_output = bias_free
        .forward(&TensorValue::constant(tensor(&INPUT_SHAPE, &INPUT_VALUES))?)?
        .value_snapshot();

Project the final feature axis

The shared forward formula is:

Y=XW+b

$W$ combines every input feature into every output feature. The optional $b$ is added to each leading position; when bias is disabled, that term is absent. The conventional module name is “linear layer,” but a nonzero bias makes the map affine rather than strictly linear.

For reverse mode, let $G=\partial L/\partial Y$ be the gradient of scalar loss $L$ with respect to $Y$ , and let $p$ range over every leading position. Then:

\begin{aligned} dX_p &= G_pW^\top, \\ dW &= \sum_p X_p^\top G_p, \\ db &= \sum_p G_p. \end{aligned}

Forward outputs remain local to each position. Because every position shares the same parameters, $dW$ and $db$ accumulate contributions across positions.

With the worked upstream seed, the exact values are:

\begin{aligned} G &= \left[ \begin{bmatrix} 1 & 0 & -1 \\ 0.5 & 2 & 1 \end{bmatrix} \right], \\ dX &= \left[ \begin{bmatrix} 2 & 1 \\ -0.5 & 3 \end{bmatrix} \right], \\ dW &= \begin{bmatrix} 0.5 & -2 & -2 \\ 3.5 & 6 & 1 \end{bmatrix}, \\ db &= \begin{bmatrix}1.5 & 2 & 0\end{bmatrix}. \end{aligned}

Keep leading axes separate from feature axes

$X$ is the input tensor with shape $[\ldots,d_{in}]$ .
$W$ is the trainable matrix with shape $[d_{in},d_{out}]$ .
$b$ is the optional width- $d_{out}$ bias, broadcast to every leading position.
$Y$ is the output tensor with shape $[\ldots,d_{out}]$ .
$d_{in}$ is the input feature width and final axis of $X$ .
$d_{out}$ is the output feature width and final axis of $Y$ .
$\ldots$ means any preserved batch, sequence, or other leading axes.

A $[d_{in},d_{out}]$ matrix changes the last width. It does not mix tokens merely because the operation is called matrix multiplication. Inputs shaped $[2]$ , $[2,2]$ , and $[1,2,2]$ therefore produce $[3]$ , $[2,3]$ , and $[1,2,3]$ with this same weight.

From adaptive responses to projections throughout a Transformer

One scalar weighted response is local arithmetic inside an adaptive system, but a language model needs vectors of hidden activations and vocabulary-wide scores at every context position; treating every output as a separate scalar unit hides the shared matrix computation.

Rosenblatt, The Perceptron: Rosenblatt describes an adaptive response architecture in which summed excitatory and inhibitory signals and reinforcement influence the selected response. This supports the early adaptive-response context, not this course’s affine formula or API.

Bengio et al. express hidden and output computation in a neural language model with trainable matrices and additive biases. The Transformer then reuses learned projections for queries, keys, values, attention outputs, position-wise feed-forward transformations, and next-token scoring.

Bengio et al., A Neural Probabilistic Language Model: Bengio et al. compute unnormalized next-word scores with $y=b+Wx+U \tanh(d+Hx)$ , making trainable matrix products and additive biases explicit inside a neural language model.

Vaswani et al., Attention Is All You Need: Vaswani et al. learn separate linear projections for queries, keys, and values, project concatenated heads again, apply two linear transformations identically at each feed-forward position, and use a learned pre-softmax projection.

A decoder applies the same learned feature projection independently at every batch and sequence position. This course keeps bias available for the historical affine form, while its target attention, SwiGLU, and vocabulary projections deliberately use the bias-free form.

The progression is the road from earlier neural computation to modern language models, not a history of programming languages. The papers do not define this course’s row orientation, errors, names, fixed seed, optional-bias API, or target bias policy.

Wrap existing differentiable operations in one named layer

Construction and forwarding report explicit parameter-rank, width, input-rank, allocation, and delegated autodiff failures:

Keep linear-layer construction and forwarding failures typed rust/crates/llm-from-scratch/src/nn/linear.rs#linear-errors

/// A rejected parameter set, input shape, allocation, or delegated tape operation.
#[derive(Clone, Debug, PartialEq)]
pub enum LinearError {
    Initialization(InitializationError),
    Autodiff(TensorAutodiffError),
    WeightRank { rank: usize },
    ZeroInputWidth,
    ZeroOutputWidth,
    BiasRank { rank: usize },
    BiasWidthMismatch { expected: usize, actual: usize },
    InputRank { rank: usize },
    InputWidthMismatch { expected: usize, actual: usize },
    BiasAllocationFailed { elements: usize },
}

impl fmt::Display for LinearError {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::Initialization(error) => error.fmt(formatter),
            Self::Autodiff(error) => error.fmt(formatter),
            Self::WeightRank { rank } => {
                write!(
                    formatter,
                    "linear weight must have rank two, got rank {rank}"
                )
            }
            Self::ZeroInputWidth => {
                formatter.write_str("linear input width must be greater than zero")
            }
            Self::ZeroOutputWidth => {
                formatter.write_str("linear output width must be greater than zero")
            }
            Self::BiasRank { rank } => {
                write!(formatter, "linear bias must have rank one, got rank {rank}")
            }
            Self::BiasWidthMismatch { expected, actual } => write!(
                formatter,
                "linear bias width must equal output width {expected}, got {actual}"
            ),
            Self::InputRank { rank } => write!(
                formatter,
                "linear input must have at least one feature axis, got rank {rank}"
            ),
            Self::InputWidthMismatch { expected, actual } => write!(
                formatter,
                "linear input final width must equal {expected}, got {actual}"
            ),
            Self::BiasAllocationFailed { elements } => write!(
                formatter,
                "could not reserve storage for {elements} linear bias values"
            ),
        }
    }
}

impl Error for LinearError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        match self {
            Self::Initialization(error) => Some(error),
            Self::Autodiff(error) => Some(error),
            _ => None,
        }
    }
}

impl From<InitializationError> for LinearError {
    fn from(error: InitializationError) -> Self {
        Self::Initialization(error)
    }
}

impl From<TensorAutodiffError> for LinearError {
    fn from(error: TensorAutodiffError) -> Self {
        Self::Autodiff(error)
    }
}

Linear owns one named $[d_{in},d_{out}]$ weight and an optional named bias. It delegates the actual matrix product and broadcast addition to the Chapter 16 autodiff operations, promotes rank-one input only around the existing matmul, and restores its vector shape afterward:

Own named projection parameters and preserve every leading axis rust/crates/llm-from-scratch/src/nn/linear.rs#linear-layer

/// One `[input_width, output_width]` feature projection with optional bias.
#[derive(Clone, Debug)]
pub struct Linear {
    parameters: NamedParameters,
    input_width: usize,
    output_width: usize,
    has_bias: bool,
}

impl Linear {
    /// Initializes a named weight and optional zero bias without partially advancing `rng`.
    pub fn new(
        parameter_prefix: impl Into<String>,
        input_width: usize,
        output_width: usize,
        with_bias: bool,
        rng: &mut SplitMix64,
    ) -> Result<Self, LinearError> {
        let parameter_prefix = parameter_prefix.into();
        let mut trial = rng.clone();
        let weight = NamedParameter::xavier_uniform(
            format!("{parameter_prefix}.weight"),
            input_width,
            output_width,
            &mut trial,
        )
        .map_err(|error| match error {
            InitializationError::ZeroFanIn => LinearError::ZeroInputWidth,
            InitializationError::ZeroFanOut => LinearError::ZeroOutputWidth,
            other => LinearError::Initialization(other),
        })?;

        let bias = if with_bias {
            let mut values = Vec::new();
            values.try_reserve_exact(output_width).map_err(|_| {
                LinearError::BiasAllocationFailed {
                    elements: output_width,
                }
            })?;
            values.resize(output_width, 0.0);
            let tensor = Tensor::from_vec(vec![output_width], values)
                .map_err(InitializationError::Tensor)?;
            Some(NamedParameter::from_tensor(
                format!("{parameter_prefix}.bias"),
                tensor,
            )?)
        } else {
            None
        };

        let layer = Self::from_parameters(weight, bias)?;
        *rng = trial;
        Ok(layer)
    }

    /// Gives layer semantics to an existing weight and optional bias.
    pub fn from_parameters(
        weight: NamedParameter,
        bias: Option<NamedParameter>,
    ) -> Result<Self, LinearError> {
        let weight_shape = weight.tensor().shape();
        if weight_shape.len() != 2 {
            return Err(LinearError::WeightRank {
                rank: weight_shape.len(),
            });
        }
        let input_width = weight_shape[0];
        let output_width = weight_shape[1];
        if input_width == 0 {
            return Err(LinearError::ZeroInputWidth);
        }
        if output_width == 0 {
            return Err(LinearError::ZeroOutputWidth);
        }

        if let Some(parameter) = &bias {
            let bias_shape = parameter.tensor().shape();
            if bias_shape.len() != 1 {
                return Err(LinearError::BiasRank {
                    rank: bias_shape.len(),
                });
            }
            if bias_shape[0] != output_width {
                return Err(LinearError::BiasWidthMismatch {
                    expected: output_width,
                    actual: bias_shape[0],
                });
            }
        }

        let has_bias = bias.is_some();
        let mut parameters = vec![weight];
        if let Some(bias) = bias {
            parameters.push(bias);
        }
        Ok(Self {
            parameters: NamedParameters::try_new(parameters)?,
            input_width,
            output_width,
            has_bias,
        })
    }

    /// Projects only the final feature axis and preserves every leading axis.
    pub fn forward(&self, input: &TensorValue) -> Result<TensorValue, LinearError> {
        let input_shape = input.shape();
        if input_shape.is_empty() {
            return Err(LinearError::InputRank { rank: 0 });
        }
        let actual_width = *input_shape.last().expect("nonempty input shape");
        if actual_width != self.input_width {
            return Err(LinearError::InputWidthMismatch {
                expected: self.input_width,
                actual: actual_width,
            });
        }

        let projected = if input_shape.len() == 1 {
            let promoted = input.reshape(&[1, self.input_width])?;
            let output = promoted.matmul(self.weight().tensor())?;
            let output = match self.bias() {
                Some(bias) => output.add(bias.tensor())?,
                None => output,
            };
            output.reshape(&[self.output_width])?
        } else {
            let output = input.matmul(self.weight().tensor())?;
            match self.bias() {
                Some(bias) => output.add(bias.tensor())?,
                None => output,
            }
        };
        Ok(projected)
    }

    pub fn weight(&self) -> &NamedParameter {
        &self.parameters.as_slice()[0]
    }

    pub fn bias(&self) -> Option<&NamedParameter> {
        self.has_bias.then(|| &self.parameters.as_slice()[1])
    }

    pub fn parameters(&self) -> &[NamedParameter] {
        self.parameters.as_slice()
    }

    pub const fn input_width(&self) -> usize {
        self.input_width
    }

    pub const fn output_width(&self) -> usize {
        self.output_width
    }

    pub const fn has_bias(&self) -> bool {
        self.has_bias
    }

    pub const fn parameter_count(&self) -> usize {
        let weight_count = self.input_width * self.output_width;
        if self.has_bias {
            weight_count + self.output_width
        } else {
            weight_count
        }
    }
}

The nonuniform reverse seed exposes all three gradient paths:

Reverse through the worked affine projection rust/demos/ch19-linear-layers/src/lib.rs#linear-gradients

    let upstream = tensor(&UPSTREAM_SHAPE, &UPSTREAM_VALUES);
    output.backward_with_seed(&upstream.view(), GraphRetention::Retain)?;
    let input_gradient = input
        .gradient_snapshot()
        .expect("trainable input stores its exact gradient");
    let weight_gradient = layer
        .weight()
        .tensor()
        .gradient_snapshot()
        .expect("trainable weight stores its exact gradient");
    let bias_gradient = layer
        .bias()
        .expect("affine fixture owns bias")
        .tensor()
        .gradient_snapshot()
        .expect("trainable bias stores its exact gradient");

Initialization composes with Chapter 17’s deterministic Xavier policy. A requested bias begins at exact zero, equal seeds reproduce weight values, and a clone deliberately keeps the same parameter leaves:

Initialize reproducibly and verify clone identity rust/demos/ch19-linear-layers/src/lib.rs#initialized-linear-layer

    let mut first_rng = SplitMix64::from_seed(19);
    let mut second_rng = SplitMix64::from_seed(19);
    let initialized = Linear::new("token_projection", 2, 3, true, &mut first_rng)?;
    let reproduced = Linear::new("token_projection", 2, 3, true, &mut second_rng)?;
    let initialized_reproducible =
        *initialized.weight().tensor().value() == *reproduced.weight().tensor().value();
    let initialized_bias_zero = initialized
        .bias()
        .expect("requested bias")
        .tensor()
        .value()
        .as_slice()
        .iter()
        .all(|&value| value == 0.0);
    let cloned = initialized.clone();
    let clone_shares_weight = initialized
        .weight()
        .tensor()
        .is_same_node(cloned.weight().tensor());
    let clone_shares_bias = initialized
        .bias()
        .expect("requested bias")
        .tensor()
        .is_same_node(cloned.bias().expect("cloned bias").tensor());

The learner report also checks vector, sequence, batch, empty-leading-axis, and error behavior:

Prepare the deterministic Chapter 19 learner report rust/demos/ch19-linear-layers/src/main.rs#learner-linear-layers-output

    let report = learner_report()?;

Run cargo run --quiet --locked -p ch19-linear-layers to inspect the complete report. It demonstrates exact forward and reverse values, rank-one shape restoration, stable parameter order and identity, safe initialization, and agreement with sampled finite differences at step $10^{-6}$ and absolute tolerance $2\times10^{-6}$ .

Trace positions, products, policy, and gradients

The Rust example records the exact values shown below: the shared parameters, each position’s products, the effect of omitting the bias, and all three reverse gradient paths.

Emit exact projection contributions, policy, axes, and gradients rust/demos/ch19-linear-layers/src/diagram_trace.rs#linear-layers-trace

pub fn render_trace() -> Result<String, Box<dyn Error>> {
    let layer = known_linear(true);
    let input = TensorValue::parameter(tensor(&INPUT_SHAPE, &INPUT_VALUES))?;
    let output = layer.forward(&input)?;
    let upstream = tensor(&UPSTREAM_SHAPE, &UPSTREAM_VALUES);
    output.backward_with_seed(&upstream.view(), GraphRetention::Retain)?;
    let input_gradient = input.gradient().expect("input gradient");
    let weight_gradient = layer.weight().tensor().gradient().expect("weight gradient");
    let bias_gradient = layer
        .bias()
        .expect("affine fixture bias")
        .tensor()
        .gradient()
        .expect("bias gradient");
    let input_value = input.value();
    let output_value = output.value();
    let weight_value = layer.weight().tensor().value();
    let bias_value = layer.bias().expect("affine fixture bias").tensor().value();
    let bias_free = known_linear(false);
    let bias_free_output = bias_free
        .forward(&TensorValue::constant(tensor(&INPUT_SHAPE, &INPUT_VALUES))?)?
        .value_snapshot();

    let mut trace = String::new();
    writeln!(trace, "TRACE linear-layers-v1 BEGIN")?;
    writeln!(
        trace,
        "FIXTURE name=known-affine-projection parameter-prefix=token_projection input-width={} output-width={} bias={} parameter-count={} input-shape={} output-shape={} upstream-shape={}",
        layer.input_width(),
        layer.output_width(),
        layer.has_bias(),
        layer.parameter_count(),
        shape(input_value.shape()),
        shape(output_value.shape()),
        shape(&UPSTREAM_SHAPE),
    )?;
    writeln!(trace, "INPUT values={}", fixed_list(input_value.as_slice()))?;
    for input_feature in 0..layer.input_width() {
        let start = input_feature * layer.output_width();
        writeln!(
            trace,
            "WEIGHT-ROW input-feature={input_feature} values={}",
            fixed_list(&weight_value.as_slice()[start..start + layer.output_width()])
        )?;
    }
    writeln!(trace, "BIAS values={}", fixed_list(bias_value.as_slice()))?;

    for position in 0..INPUT_SHAPE[1] {
        let input_start = position * INPUT_SHAPE[2];
        let output_start = position * layer.output_width();
        let position_input =
            &input_value.as_slice()[input_start..input_start + layer.input_width()];
        for (output_feature, &bias) in bias_value.as_slice().iter().enumerate() {
            let result = output_value.as_slice()[output_start + output_feature];
            writeln!(
                trace,
                "CELL position={position} coordinate=0,{position} output-feature={output_feature} input={} products={} weighted-sum={} bias={} result={}",
                fixed_list(position_input),
                products(
                    position_input,
                    output_feature,
                    weight_value.as_slice(),
                    layer.output_width(),
                ),
                fixed(result - bias),
                fixed(bias),
                fixed(result)
            )?;
        }
        writeln!(
            trace,
            "POSITION-GRADIENT position={position} coordinate=0,{position} upstream={} input-gradient={}",
            fixed_list(&UPSTREAM_VALUES[output_start..output_start + layer.output_width()]),
            fixed_list(&input_gradient.as_slice()[input_start..input_start + layer.input_width()])
        )?;
    }

    writeln!(
        trace,
        "WEIGHT-GRADIENT shape={} values={}",
        shape(weight_gradient.shape()),
        fixed_list(weight_gradient.as_slice())
    )?;
    writeln!(
        trace,
        "BIAS-GRADIENT shape={} values={}",
        shape(bias_gradient.shape()),
        fixed_list(bias_gradient.as_slice())
    )?;
    writeln!(
        trace,
        "POLICY affine-parameters={} bias-free-parameters={} bias-free-output={}",
        layer.parameter_count(),
        bias_free.parameter_count(),
        fixed_list(bias_free_output.as_slice())
    )?;
    writeln!(
        trace,
        "AXES input-leading={} output-leading={} preserved=true mixed-axis=feature",
        shape(&input_value.shape()[..input_value.shape().len() - 1]),
        shape(&output_value.shape()[..output_value.shape().len() - 1]),
    )?;
    writeln!(trace, "TRACE linear-layers-v1 END")?;
    Ok(trace)
}

Follow one shared projection across two token positions

Read exact Rust-authored shapes, shared weights, per-output contributions, affine and bias-free results, and gradients accumulated across both positions.

Named parameter: token_projection.weight
Input width: $2$
Output width: $3$
Bias: Enabled
Parameter scalars: $9$
Input shape: $\left[1,2,2\right]$
Output shape: $\left[1,2,3\right]$
Upstream shape: $\left[1,2,3\right]$

Preserve positions; change feature width

The leading batch and sequence coordinates are unchanged. Only the final feature axis grows from two coordinates to three.

Input vector

\left[1,2,2\right]

Leading axes preserved

Output

\left[1,2,3\right]

Only the feature axis is mixed $\left[1,2\right]=\left[1,2\right]$

Share one weight matrix and bias

Parameter	Value
$W$	$\begin{bmatrix}1&0&-1\\2&0.5&1\end{bmatrix}$
Bias	$\left[0.5,-0.5,1\right]$

Project each position independently

Position	Leading coordinate	Input vector	Output
$0$	$\left(0,0\right)$	$\left[1,2\right]$	$\left[5.5,0.5,2\right]$
$1$	$\left(0,1\right)$	$\left[-1,3\right]$	$\left[5.5,1,5\right]$

Expand one output coordinate

This first output coordinate uses both input features. Their products form the weighted sum, then the matching bias produces the result.

Leading coordinate: $\left(0,0\right),\;y_{0}$
Result: $\begin{aligned}y_{0}&=1\cdot1+2\cdot2+0.5\\&=5+0.5=5.5\end{aligned}$

Compare affine and bias-free policies

The affine path adds one shared bias coordinate per output feature. The target decoder chooses the bias-free path for attention, SwiGLU, and vocabulary projections.

Position	Output
$\left(0,0\right)$	Affine projection Parameter scalars: $9$ $\left[5.5,0.5,2\right]$ Bias-free projection Parameter scalars: $6$ $\left[5,1,1\right]$
$\left(0,1\right)$	Affine projection Parameter scalars: $9$ $\left[5.5,1,5\right]$ Bias-free projection Parameter scalars: $6$ $\left[5,1.5,4\right]$

Accumulate gradients for shared parameters

Each input position receives its own gradient. The shared weight and bias collect contributions from both positions.

Per-position gradients
$p$	$(b,t)$	$G_p$	$dX_p$
$dX_{0}$	$\left(0,0\right)$	$\left[1,0,-1\right]$	$\left[2,1\right]$
$dX_{1}$	$\left(0,1\right)$	$\left[0.5,2,1\right]$	$\left[-0.5,3\right]$

Shared-parameter gradients
$\theta$	$\sum_p$	$d\theta$
$dW$	$\sum_p X_p^\top G_p$	$\left[2,3\right]$ $\begin{bmatrix}0.5&-2&-2\\3.5&6&1\end{bmatrix}$
$db$	$\sum_p G_p$	$\left[3\right]$ $\left[1.5,2,0\right]$

Read the position rows first to see that the projection never crosses a leading coordinate. Then compare the affine and bias-free outputs and finish with the local input gradients and the shared parameter sums.

Predict before checking the executable evidence

Predict the output shape for $[4,7,2]$ through a $[2,3]$ weight.
For the worked layer, predict the affine output for $[0,0]$ and the bias-free output.
Decide whether changing one token vector can change another position’s output in this layer.
For a separate projection with $d_{out}=2$ , predict $db$ when three positions receive $[1,0]$ , $[0,2]$ , and $[3,4]$ .
Predict the parameter counts for $[2,3]$ projections with and without bias.
Decide whether $[2,0]$ is a valid weight and whether $[0,2]$ is a valid input to a $[2,3]$ layer.
Explain why calling the biased map strictly linear is imprecise.
Identify which source supports affine computation inside a language model and which implementation policies none of the papers define.

Check the predictions

The output shape is $[4,7,3]$ .
The affine output is $[0.5,-0.5,1]$ ; disabling bias produces $[0,0,0]$ .
No. The shared matrix acts independently at every leading coordinate.
The bias gradient is the position sum $[4,6]$ .
The affine layer owns nine scalars; the bias-free layer owns six.
Zero output width is invalid; $[0,2]$ has an empty leading axis and valid final width, so it becomes $[0,3]$ .
A nonzero additive bias means the map does not preserve the origin, so it is affine.
Bengio et al. support affine computation inside a neural language model. Orientation, errors, names, seed, API, and target bias policy are course choices.

Hand reusable projections to the first gated block

The cumulative model now owns a named differentiable projection from $[\ldots,d_{in}]$ to $[\ldots,d_{out}]$ . The same abstraction will create attention queries, keys, values, attention outputs, feed-forward branches, and vocabulary scores. The target decoder chooses the bias-free policy for those paths.

Stacking bias-free projections alone still collapses to one projection: $(XW_1)W_2=X(W_1W_2)$ . Chapter 20 inserts a nonlinear SiLU gate, so that collapse no longer applies.