24 · Content revision 3

Keep an identity path around each learned update

Trace exact-shape residual addition, its identity and learned gradient paths, zero-branch learning, and repeated plain versus residual transformations.

Predict what the identity path preserves

Start with $x=[2,-1]$ and a bias-free square linear branch whose output is $F(x)=[-1,-2.25]$ . Predict $y=x+F(x)$ before running the example. Elementwise addition gives $[1,-3.25]$ and preserves shape $[2]$ .

Now set the branch weights to zero. The branch is then identically zero, not merely zero at this one input. Predict three things for upstream $\bar y=[1,1]$ : the output, the input gradient, and the branch weight gradient. The first two follow the identity path exactly. The weight gradient can still be nonzero because changing a zero weight would change the branch output.

Add a learned update to the unchanged stream

The entire residual connection is:

y=x+F(x)

Both terms must have the same complete shape:

\operatorname{shape}(F(x))=\operatorname{shape}(x)=\operatorname{shape}(y)

That is stricter than ordinary broadcasting. A branch shaped $[2]$ may be a valid generic addend for an input shaped $[2,2]$ , but it is not the same residual stream and this chapter rejects the merge.

Reverse mode follows the two parent edges and accumulates their contributions:

\bar{x}=\bar{y}+J_F(x)^\top\bar{y}

The identity path contributes $\bar y$ directly. The branch contributes its vector-Jacobian product $J_F(x)^\top\bar y$ . Either contribution can reinforce, cancel, or amplify the other, so an identity term is not a guarantee that every gradient remains large or well behaved.

A scaled form can be useful for intuition:

y=x+\alpha F(x)

At $\alpha=0$ only the identity forward term remains; at $\alpha=1$ this is the ordinary residual connection implemented here. The cumulative decoder does not adopt a separate residual-scaling policy in this chapter.

Keep the stream and update roles separate

$x$ is the input tensor carried unchanged by the identity path.
$F$ is the learned branch mapping; it may later be attention or a feed-forward network.
$F(x)$ is the branch update and has exactly the same shape as $x$ .
$y$ is their elementwise sum and retains that shape.
$\bar y$ is the upstream output adjoint and $\bar x$ is the accumulated input adjoint.
$J_F(x)^\top\bar y$ is the branch’s reverse-mode contribution.
$\alpha$ is an optional explanatory scale, not a trainable value in this implementation.

Addition is not concatenation: no feature axis grows. It is also not normalization: no mean, variance, or root-mean-square is computed.

From deep plain transformations to the Transformer residual stream

He et al. observed a degradation problem in which deeper plain networks could have higher training error even though the added layers should in principle be able to represent identity mappings.

The earlier primary source is He et al., Deep Residual Learning for Image Recognition. He et al. report the deep-plain-network degradation problem and reformulate a same-dimensional block as a learned residual function plus a parameter-free identity shortcut. Residual learning made the identity route explicit, and the Transformer later placed residual additions around every attention and feed-forward sublayer at one common model width.

The later primary source is Vaswani et al., Attention Is All You Need. Vaswani et al. place a residual connection around every encoder and decoder sublayer and keep sublayer outputs at a common model width required by the addition. The original Transformer follows each sum with LayerNorm. This course later assembles a pre-RMSNorm decoder, so the paper is not evidence for that later ordering.

A decoder-only Transformer maintains a residual stream across its stack while learned attention and feed-forward branches contribute same-shaped updates; later chapters add normalization and assemble those branches.

Residual addition gives every same-width Transformer sublayer an explicit identity route alongside its learned update. The route preserves a direct forward contribution and adds a direct reverse-mode gradient contribution, without guaranteeing that every deep model will train successfully.

Residual learning made the identity shortcut explicit in deep vision networks. Transformer architectures then reused that mechanism around attention and feed-forward sublayers. The four-layer comparison isolates one architectural consequence: each ordinary layer keeps only the learned transformation, whereas each residual layer adds that transformation to an explicit identity path:

Compare repeated plain and residual transformations rust/demos/ch24-residual-connections/src/lib.rs#residual-stack

fn stack_fixture(use_residual: bool) -> Result<StackFixture, Box<dyn Error>> {
    let prefix = if use_residual { "residual" } else { "plain" };
    let input = TensorValue::parameter(tensor(&INPUT_SHAPE, &INPUT_VALUES))?;
    let mut current = input.clone();
    let mut values = vec![current.value_snapshot()];
    let mut layers = Vec::new();
    for depth in 0..4 {
        let layer = named_linear(
            format!("{prefix}.stack.{depth}.branch.weight"),
            &STACK_WEIGHT_VALUES,
        )?;
        let branch = layer.forward(&current)?;
        current = if use_residual {
            residual_add(&current, &branch)?
        } else {
            branch
        };
        values.push(current.value_snapshot());
        layers.push(layer);
    }
    current.backward_with_seed(
        &tensor(&INPUT_SHAPE, &UPSTREAM_VALUES).view(),
        GraphRetention::Retain,
    )?;
    let parameter_names = layers
        .iter()
        .map(|layer| layer.weight().name().to_owned())
        .collect();
    let parameter_gradients_finite_nonzero = layers.iter().all(|layer| {
        layer.weight().tensor().gradient().is_some_and(|gradient| {
            gradient.as_slice().iter().all(|value| value.is_finite())
                && gradient.as_slice().iter().any(|value| *value != 0.0)
        })
    });
    Ok(StackFixture {
        values,
        input_gradient: input.gradient_snapshot().expect("stack input gradient"),
        parameter_names,
        parameter_gradients_finite_nonzero,
    })
}

Reject broadcasting before adding the paths

The reusable utility owns no parameters. It first compares complete shapes, then delegates a valid merge to the cumulative differentiable addition. This keeps both operand edges and therefore both reverse contributions:

Enforce exact shape equality and preserve both tape edges rust/crates/llm-from-scratch/src/nn/residual.rs#residual-add

/// Adds an identity path to one same-shaped branch output.
///
/// The lower-level tensor addition supports broadcasting. A residual connection
/// does not: the branch must return exactly the residual stream's complete shape.
/// This utility owns no parameters and preserves both operands' tape edges.
pub fn residual_add(
    identity: &TensorValue,
    branch_output: &TensorValue,
) -> Result<TensorValue, ResidualError> {
    let identity_shape = identity.shape();
    let branch_shape = branch_output.shape();
    if identity_shape != branch_shape {
        return Err(ResidualError::ShapeMismatch {
            identity: identity_shape,
            branch: branch_shape,
        });
    }
    Ok(identity.add(branch_output)?)
}

The fixed-value square Linear fixture owns residual.branch.weight. Its known forward pass produces $F(x)=[-1,-2.25]$ . For $\bar y=[1,1]$ , the identity contribution is $[1,1]$ , the branch contribution is $[-0.5,2.25]$ , and their sum is $\bar x=[0.5,3.25]$ . The branch’s matrix gradient is $[2,2,-1,-1]$ in row-major order:

Expose known residual and zero-branch gradients rust/demos/ch24-residual-connections/src/lib.rs#residual-fixture

fn primary_fixture() -> Result<PrimaryFixture, Box<dyn Error>> {
    let layer = named_linear("residual.branch.weight", &WEIGHT_VALUES)?;
    let input = TensorValue::parameter(tensor(&INPUT_SHAPE, &INPUT_VALUES))?;
    let branch_output = layer.forward(&input)?;
    let output = residual_add(&input, &branch_output)?;
    let upstream = tensor(&INPUT_SHAPE, &UPSTREAM_VALUES);
    output.backward_with_seed(&upstream.view(), GraphRetention::Retain)?;

    let branch_probe_layer = named_linear("residual.branch_probe.weight", &WEIGHT_VALUES)?;
    let branch_probe_input = TensorValue::parameter(tensor(&INPUT_SHAPE, &INPUT_VALUES))?;
    branch_probe_layer
        .forward(&branch_probe_input)?
        .backward_with_seed(&upstream.view(), GraphRetention::Retain)?;

    Ok(PrimaryFixture {
        input: input.value_snapshot(),
        branch_parameter_name: layer.weight().name().to_owned(),
        branch_weight: layer.weight().tensor().value_snapshot(),
        branch_output: branch_output.value_snapshot(),
        residual_output: output.value_snapshot(),
        upstream: upstream.clone(),
        identity_gradient: upstream,
        branch_input_gradient: branch_probe_input
            .gradient_snapshot()
            .expect("branch probe input gradient"),
        input_gradient: input.gradient_snapshot().expect("residual input gradient"),
        weight_gradient: layer
            .weight()
            .tensor()
            .gradient_snapshot()
            .expect("residual branch weight gradient"),
    })
}

fn zero_branch_fixture() -> Result<(Tensor, Tensor, Tensor, bool), Box<dyn Error>> {
    let layer = named_linear("residual.zero.weight", &ZERO_WEIGHT_VALUES)?;
    let input = TensorValue::parameter(tensor(&INPUT_SHAPE, &INPUT_VALUES))?;
    let branch_output = layer.forward(&input)?;
    let output = residual_add(&input, &branch_output)?;
    output.backward_with_seed(
        &tensor(&INPUT_SHAPE, &UPSTREAM_VALUES).view(),
        GraphRetention::Retain,
    )?;
    let weight_gradient = layer
        .weight()
        .tensor()
        .gradient_snapshot()
        .expect("zero branch weight gradient");
    let nonzero = weight_gradient
        .as_slice()
        .iter()
        .all(|value| value.is_finite())
        && weight_gradient.as_slice().iter().any(|value| *value != 0.0);
    Ok((
        output.value_snapshot(),
        input
            .gradient_snapshot()
            .expect("zero branch input gradient"),
        weight_gradient,
        nonzero,
    ))
}

The zero-weight branch produces $F(x)=0$ for every input and therefore contributes zero to this input gradient. Its weight gradient remains $[2,2,-1,-1]$ . Do not generalize from a branch output that happens to be zero at one point: $F(x)=0$ alone does not imply that $J_F(x)=0$ .

The numeric check evaluates an independent raw-tensor implementation of the scalar objective $\sum_i(x+xW)_i^2$ at each perturbed coordinate, then checks all two input coordinates plus all four weight coordinates:

Check input and branch-weight gradients numerically rust/demos/ch24-residual-connections/src/lib.rs#residual-gradcheck

fn numeric_gradient_fixture() -> Result<(usize, usize, bool), Box<dyn Error>> {
    let layer = named_linear("residual.gradcheck.weight", &WEIGHT_VALUES)?;
    let input = TensorValue::parameter(tensor(&INPUT_SHAPE, &INPUT_VALUES))?;
    let branch = layer.forward(&input)?;
    let output = residual_add(&input, &branch)?;
    output.mul(&output)?.sum_axis(0, false)?.backward()?;
    let analytic_input = input.gradient().expect("analytic input gradient");
    let analytic_weight = layer
        .weight()
        .tensor()
        .gradient()
        .expect("analytic weight gradient");

    let frozen_weight = tensor(&WEIGHT_SHAPE, &WEIGHT_VALUES);
    let mut input_probe = tensor(&INPUT_SHAPE, &INPUT_VALUES);
    let input_check = sampled_tensor_gradient_check(
        &mut input_probe,
        &analytic_input.view(),
        GRADCHECK_STEP,
        GRADCHECK_TOLERANCE,
        2,
        |probe| squared_residual_objective(probe, &frozen_weight),
    )?;

    let frozen_input = tensor(&INPUT_SHAPE, &INPUT_VALUES);
    let mut weight_probe = tensor(&WEIGHT_SHAPE, &WEIGHT_VALUES);
    let weight_check = sampled_tensor_gradient_check(
        &mut weight_probe,
        &analytic_weight.view(),
        GRADCHECK_STEP,
        GRADCHECK_TOLERANCE,
        4,
        |probe| squared_residual_objective(&frozen_input, probe),
    )?;

    Ok((
        input_check.checks.len(),
        weight_check.checks.len(),
        input_check.passed && weight_check.passed,
    ))
}

The report renderer consumes the checked fixture and prints deterministic text:

Render the complete Chapter 24 learner report rust/demos/ch24-residual-connections/src/lib.rs#learner-residual-report

pub fn render_learner_report() -> Result<String, Box<dyn Error>> {
    let report = learner_report()?;
    let mut output = String::new();
    writeln!(output, "chapter=24-residual-connections")?;
    writeln!(
        output,
        "prediction=zero branch preserves output and input gradient but its weight gradient can be nonzero"
    )?;
    writeln!(
        output,
        "input=shape:{} values:{}",
        bracketed_shape(report.input.shape()),
        bracketed_values(&report.input)
    )?;
    writeln!(
        output,
        "branch_parameter=name:{} shape:{} values:{}",
        report.branch_parameter_name,
        bracketed_shape(report.branch_weight.shape()),
        bracketed_values(&report.branch_weight)
    )?;
    writeln!(
        output,
        "branch_output=shape:{} values:{}",
        bracketed_shape(report.branch_output.shape()),
        bracketed_values(&report.branch_output)
    )?;
    writeln!(
        output,
        "residual_output=shape:{} values:{}",
        bracketed_shape(report.residual_output.shape()),
        bracketed_values(&report.residual_output)
    )?;
    writeln!(
        output,
        "upstream=shape:{} values:{}",
        bracketed_shape(report.upstream.shape()),
        bracketed_values(&report.upstream)
    )?;
    writeln!(
        output,
        "identity_gradient={}",
        bracketed_values(&report.identity_gradient)
    )?;
    writeln!(
        output,
        "branch_input_gradient={}",
        bracketed_values(&report.branch_input_gradient)
    )?;
    writeln!(
        output,
        "input_gradient={}",
        bracketed_values(&report.input_gradient)
    )?;
    writeln!(
        output,
        "weight_gradient=shape:{} values:{}",
        bracketed_shape(report.weight_gradient.shape()),
        bracketed_values(&report.weight_gradient)
    )?;
    writeln!(
        output,
        "zero_branch=output:{} input_gradient:{} weight_gradient_nonzero:{}",
        bracketed_values(&report.zero_output),
        bracketed_values(&report.zero_input_gradient),
        report.zero_weight_gradient_nonzero
    )?;
    writeln!(
        output,
        "shape_error=identity:{} branch:{} broadcastable:{} rejected:{}",
        bracketed_shape(&report.mismatch_identity_shape),
        bracketed_shape(&report.mismatch_branch_shape),
        report.generic_add_broadcasts,
        report.residual_mismatch_rejected
    )?;
    for row in &report.stack {
        writeln!(
            output,
            "stack[{}]=plain:{} residual:{}",
            row.depth,
            bracketed_values(&row.plain),
            bracketed_values(&row.residual)
        )?;
    }
    writeln!(
        output,
        "stack_input_gradients=plain:{} residual:{}",
        bracketed_values(&report.plain_stack_input_gradient),
        bracketed_values(&report.residual_stack_input_gradient)
    )?;
    writeln!(
        output,
        "stack_parameters={}",
        report.stack_parameter_names.join(",")
    )?;
    writeln!(
        output,
        "numeric_gradient=input_checks:{} weight_checks:{} tolerance:{:.6} passed:{}",
        report.input_gradient_checks,
        report.weight_gradient_checks,
        GRADCHECK_TOLERANCE,
        report.numeric_gradient_passed
    )?;
    writeln!(
        output,
        "historical=plain_depth4_retention:{} residual_depth4_retention:{}",
        fixed(report.plain_stack_input_gradient.as_slice()[0]),
        fixed(report.residual_stack_input_gradient.as_slice()[0])
    )?;
    writeln!(
        output,
        "same_fixture_replays_bitwise={}",
        report.same_fixture_replays_bitwise
    )?;
    writeln!(
        output,
        "next=normalize each residual branch input with RMSNorm"
    )?;
    Ok(output)
}

Print the frozen learner report rust/demos/ch24-residual-connections/src/main.rs

use std::error::Error;

use ch24_residual_connections::render_learner_report;

fn main() -> Result<(), Box<dyn Error>> {
    print!("{}", render_learner_report()?);
    Ok(())
}

Run cargo run --quiet --locked -p ch24-residual-connections. Its stdout is frozen in rust/demos/ch24-residual-connections/expected.txt.

Follow both paths without hiding the merge

The trace supplies every vector, shape decision, stack row, gradient check, and verified residual property shown below. Read the two rows as simultaneous routes: the identity value passes unchanged, the learned branch produces an update, and the merge adds them coordinate by coordinate:

Emit exact forward, reverse, stack, and proof records rust/demos/ch24-residual-connections/src/diagram_trace.rs#residual-connections-trace

/// Renders the exact Rust-owned evidence consumed by the static chapter diagram.
pub fn render_trace() -> Result<String, Box<dyn Error>> {
    let report = learner_report()?;
    if report.stack.len() != 5 {
        return Err("residual trace requires depths zero through four".into());
    }
    let mut trace = String::new();
    writeln!(trace, "TRACE residual-connections-v1 BEGIN")?;
    writeln!(
        trace,
        "CONFIG name=known-residual-linear shape={} branch-parameter={}",
        x_shape(report.input.shape()),
        report.branch_parameter_name
    )?;
    writeln!(
        trace,
        "FORWARD input={} branch={} output={}",
        bracketed_values(&report.input),
        bracketed_values(&report.branch_output),
        bracketed_values(&report.residual_output)
    )?;
    writeln!(
        trace,
        "BACKWARD upstream={} identity={} branch={} input={}",
        bracketed_values(&report.upstream),
        bracketed_values(&report.identity_gradient),
        bracketed_values(&report.branch_input_gradient),
        bracketed_values(&report.input_gradient)
    )?;
    writeln!(
        trace,
        "PARAMETER name={} shape={} gradient={}",
        report.branch_parameter_name,
        x_shape(report.weight_gradient.shape()),
        bracketed_values(&report.weight_gradient)
    )?;
    writeln!(
        trace,
        "ZERO-BRANCH output={} input-gradient={} weight-gradient={} weight-gradient-nonzero={}",
        bracketed_values(&report.zero_output),
        bracketed_values(&report.zero_input_gradient),
        bracketed_values(&report.zero_weight_gradient),
        report.zero_weight_gradient_nonzero
    )?;
    writeln!(
        trace,
        "SHAPE-ERROR identity={} branch={} broadcastable={} rejected={}",
        bracketed_shape(&report.mismatch_identity_shape),
        bracketed_shape(&report.mismatch_branch_shape),
        report.generic_add_broadcasts,
        report.residual_mismatch_rejected
    )?;
    for row in &report.stack {
        writeln!(
            trace,
            "STACK depth={} plain={} residual={}",
            row.depth,
            bracketed_values(&row.plain),
            bracketed_values(&row.residual)
        )?;
    }
    writeln!(
        trace,
        "STACK-GRADIENT plain={} residual={} parameters={}",
        bracketed_values(&report.plain_stack_input_gradient),
        bracketed_values(&report.residual_stack_input_gradient),
        report.stack_parameter_names.join(",")
    )?;
    writeln!(
        trace,
        "GRADCHECK input-checks={} weight-checks={} tolerance={:.6} passed={}",
        report.input_gradient_checks,
        report.weight_gradient_checks,
        GRADCHECK_TOLERANCE,
        report.numeric_gradient_passed
    )?;
    writeln!(
        trace,
        "PROOF identity=exact gradient=added parameters=branch-owned broadcast=forbidden"
    )?;
    writeln!(trace, "TRACE residual-connections-v1 END")?;
    Ok(trace)
}

Follow the identity path and learned update

Trace exact fixture values through the forward merge and both reverse contributions, then compare zero-branch, shape-error, and four-layer stack evidence.

Split and rejoin in the forward pass

Solid identity path
Dashed learned path
Double-border merge

Input

x=[2.000000,-1.000000]

Identity path

x=[2.000000,-1.000000]

Learned branch

F(x)=[-1.000000,-2.250000]

Add

+

Residual output

y=[1.000000,-3.250000]

Exact same-shaped forward values

Add both reverse-mode contributions

Solid identity path
Dashed learned path
Double-border merge

Upstream gradient

\bar y=[1.000000,1.000000]

Identity contribution

\bar y=[1.000000,1.000000]

Branch contribution

J_F(x)^\top\bar y=[-0.500000,2.250000]

Add

+

Input gradient

\bar x=[0.500000,3.250000]

The identity term and branch vector-Jacobian product add

Check parameters, identity, and shape

Branch-owned parameters

residual.branch.weight

$W\in\mathbb{R}^{2\times2}$

Weight gradient

$\bar W=[2.000000,2.000000,-1.000000,-1.000000]$

Zero-weight branch

$F(x)=0,\quad y=x=[2.000000,-1.000000]$

$\bar x=\bar y=[1.000000,1.000000]$

$\bar W=[2.000000,2.000000,-1.000000,-1.000000]\ne0$

The trace shows identity forward behavior and a total input gradient equal to the upstream gradient, while the branch weight gradient remains nonzero.

Broadcastable is still invalid

Residual paths require exact equality

$[2,2]\ne[2]$

Generic addition: Accepted broadcastable=true
Residual merge: Rejected rejected=true

Sampled gradient check

$n_x=2$ $n_W=4$ $\tau=0.000002$

Accepted passed=true

Verified residual properties

Fixed-value fixture known-residual-linear

$\operatorname{shape}(x)=[2]$

Branch-owned parameters residual.branch.weight

identity=exact

gradient=added

parameters=branch-owned

broadcast=forbidden

Compare repeated plain and residual transformations

Exact values at every depth
Depth	Plain stack	Residual stack
$0$	$[2.000000,-1.000000]$	$[2.000000,-1.000000]$
$1$	$[-0.500000,0.250000]$	$[1.500000,-0.750000]$
$2$	$[0.125000,-0.062500]$	$[1.125000,-0.562500]$
$3$	$[-0.031250,0.015625]$	$[0.843750,-0.421875]$
$4$	$[0.007812,-0.003906]$	$[0.632812,-0.316406]$
Input gradient	$[0.003906,0.003906]$	$[0.316406,0.316406]$

residual.stack.0.branch.weight
residual.stack.1.branch.weight
residual.stack.2.branch.weight
residual.stack.3.branch.weight

The solid upper route carries the identity value, the dashed lower route carries the learned update, and both meet at the double-bordered addition. The backward flow uses the same split to show why the two gradient contributions accumulate.

The four diagonal branch matrices each multiply their input by $-0.25$ . The plain depth-four input gradient retains a factor of $0.00390625$ ; the residual stack retains $0.31640625$ . These exact toy values show what the explicit identity terms change in this fixture. They are not a universal optimization bound.

Predict before following the trace

Compute $y$ from the given $x$ and $F(x)$ .
Split $\bar x$ into its identity and branch contributions.
Predict the zero-weight branch’s output, input gradient, and weight gradient.
Decide whether shapes $[2,2]$ and $[2]$ form a valid residual connection.
Predict the gradient if both operands of the merge are the same tracked tensor.
Compute the depth-four plain and residual multipliers for branch factor $-0.25$ .
Explain why residual addition neither concatenates nor normalizes features.
Decide whether the identity path guarantees successful deep-model training.

Check the predictions

$y=[1,-3.25]$ .
$\bar x=[1,1]+[-0.5,2.25]=[0.5,3.25]$ .
The output is $[2,-1]$ , the input gradient is $[1,1]$ , and the weight gradient is $[2,2,-1,-1]$ , which is nonzero.
No. Generic addition broadcasts them, but a residual merge requires exact shape equality.
Two distinct operand edges reach the same leaf, so it receives twice the upstream gradient.
The plain multiplier is $(-0.25)^4=0.00390625$ ; the residual multiplier is $(1-0.25)^4=0.31640625$ .
Addition preserves the feature width and computes no normalization statistic.
No. The identity term supplies a direct path, but the branch Jacobian can still cancel, amplify, or destabilize the total gradient.

Normalize the branch input next

The cumulative decoder now has an exact-shape merge that will carry one residual stream around learned sublayers. Chapter 25 normalizes the values entering each branch while the identity path bypasses that normalization operation.

The current branch is a tiny square linear map chosen for transparent arithmetic. Later attention and SwiGLU branches will use the same merge invariant: return a tensor at model width, then add it to the unchanged residual stream.