14 · Content revision 6

Accumulate gradients through a scalar graph

Build reverse-mode scalar autodiff in Rust, accumulate gradients across reused graph edges, and verify them for LLM training.

Predict a reused scalar’s gradient

Start with one tracked scalar. The multiplication uses x twice, and the addition uses square twice:

x = 2
square = x * x
loss = square + square

The forward prediction is $\mathrm{square}=4$ and $\mathrm{loss}=8$ . Now seed the scalar output with $\bar{\mathrm{loss}}=1$ . Addition has local derivative $1$ for each operand. Both operands are the same square node, but they remain two uses, so the reverse contributions are $1$ and $1$ , giving $\bar{\mathrm{square}}=2$ .

The multiplication also has two operand uses of the same $x$ node. Each local derivative is the other operand’s primal value $2$ . Multiplying by the upstream adjoint $2$ gives contributions $4$ and $4$ , so $\bar{x}=8$ .

Construct one three-node graph while retaining four repeated operand edges rust/demos/ch14-scalar-autodiff/src/lib.rs#shared-scalar-fixture

/// Builds one shared DAG, runs it twice, clears it, and runs one fresh pass.
pub fn reused_square_example() -> Result<ReusedSquareExample, ScalarAutodiffError> {
    let x = Scalar::variable(REUSED_INPUT)?;
    let square = x.mul(&x)?;
    let loss = square.add(&square)?;

    let first_pass = loss.backward()?;
    let first = snapshot(&x, &square, &loss);
    loss.backward()?;
    let repeated = snapshot(&x, &square, &loss);
    loss.zero_grad();
    let zeroed = snapshot(&x, &square, &loss);
    loss.backward()?;
    let after_zero = snapshot(&x, &square, &loss);

    Ok(ReusedSquareExample {
        x_value: x.value(),
        square_value: square.value(),
        loss_value: loss.value(),
        first_pass,
        first,
        repeated,
        zeroed,
        after_zero,
    })
}

Accumulate every reverse path

For one fresh reverse pass, the exact rule combines the output boundary with edge-based accumulation:

\bar v=s\,\mathbf{1}[v=o]+\sum_{e\in E_o(v)}\bar{c(e)}\,d_e

Here $o$ is the selected tracked scalar output and $s$ is the explicit finite seed supplied by the caller. The method backward_with_seed installs $s$ at $o$ ; backward() uses $s=1$ . The node $v$ is tracked and belongs to the dependency subgraph traversed backward from $o$ . The indicator is $1$ when $v$ and $o$ are the same graph node and $0$ otherwise; two nodes with equal primal values do not pass that identity test. It therefore installs $\bar o=s$ even though the selected output has no consuming edge in its own backward subgraph. $E_o(v)$ contains one distinct edge for every occurrence of tracked $v$ as an operand inside that subgraph; a consumer unrelated to $o$ is excluded. Untracked constants and detached values can still appear in the structural traversal, but they are outside the active adjoint recurrence: no pass-local adjoint is stored for them and propagation into them is skipped. The result $c(e)$ consumes that occurrence, and $d_e$ is the slot-local derivative evaluated from stored forward values. This is the chain rule with its output boundary and branching made explicit. In $x\cdot x$ , the two operand slots produce two edges even though both point to the same $x$ node. Each edge contributes an upstream adjoint times its own local derivative; addition combines those contributions. Assignment would silently discard all but one.

The graph is evaluated forward before this rule runs. A result-last topological list is traversed in reverse, so all contributions have reached a consuming result before it sends its completed pass-local adjoint to its operand values. Accumulation across separate backward calls is a second operation: $g_v\mathrel{+}=\bar v$ only after the fresh pass has been validated.

Name the graph and adjoints

Symbol	Operational meaning
$v$	One tracked scalar node in the dependency subgraph traversed backward from $o$ .
$\bar v$	The pass-local adjoint under seed $s$ : $s$ times the derivative of selected output $o$ with respect to $v$ .
$o$	The selected tracked scalar output whose reverse pass is being evaluated.
$s$	The finite scalar seed supplied by the caller and installed at $o$ ; `backward()` uses $s=1$ .
$\mathbf{1}[v=o]$	The graph-node identity indicator: $1$ when $v$ is output node $o$ , otherwise $0$ , even if another node has the same primal value.
$e$	One distinct outgoing edge for one occurrence of $v$ as an operand.
$E_o(v)$	The distinct operand-use edges leaving tracked $v$ inside $o$ ‘s reverse-pass dependency subgraph.
$c(e)$	The result node that consumes the operand occurrence represented by $e$ .
$\bar{c(e)}$	The pass-local adjoint already accumulated at that consuming result.
$d_e$	The result’s local derivative with respect to this operand occurrence, evaluated from stored forward values.

The topology contains x, square, and loss once each, yet $\mathrm{square}=x\cdot x$ stores two edges to x and $\mathrm{loss}=\mathrm{square}+\mathrm{square}$ stores two edges to square. Deduplicating node visits is correct; deduplicating those four operand uses is not.

From next-word updates to scaled autoregressive Transformers

Bengio et al.’s neural language model learns next-word probabilities and distributed word features with an explicit forward phase followed by equations that propagate gradients and update output-layer, hidden-layer, and word-representation parameters. Baydin et al. show that symbolic differentiation can duplicate shared expressions, while forward mode needs one sweep per independent input variable or direction to recover a scalar loss’s full gradient. Both become unwieldy for a language model with many parameters.

An early neural-language-model example is Bengio et al., A Neural Probabilistic Language Model. Bengio et al. learn next-word probabilities and word-feature parameters and publish a forward phase plus a backward/update phase that clears and adds gradients through output units, hidden units, and input word features.

Baydin et al. describe reverse mode as recording dependencies during a forward evaluation and propagating adjoints from one scalar output back through the graph, adding contributions from every path. That direction fits a scalar training objective with many parameters. Vaswani et al. then train repeated Transformer attention and feed-forward layers, and Radford et al. scale autoregressive Transformer language models from 12 to 48 layers and from 117 million to 1.542 billion parameters.

The reverse-mode account is Baydin et al., Automatic Differentiation in Machine Learning: a Survey. Baydin et al. show how symbolic differentiation can duplicate shared expressions, explain that forward mode needs one sweep per independent input variable or direction for a scalar output’s full gradient, and describe reverse dependency recording and adjoint accumulation in one reverse pass.

The Transformer training example is Vaswani et al., Attention Is All You Need. Vaswani et al. build the Transformer from repeated attention and position-wise feed-forward sublayers and train base models for 100,000 steps and big models for 300,000 steps with Adam.

The scaled autoregressive-language-model example is Radford et al., Language Models are Unsupervised Multitask Learners. Radford et al. use Transformer-based autoregressive language models and report four sizes spanning 12 to 48 layers and 117 million to 1.542 billion parameters.

This chapter isolates reverse accumulation in a tiny scalar graph, then checks selected derivatives against Chapter 13’s materially separate sampled numerical cross-check. Both paths still use the same mathematical function and input value and share IEEE f64 arithmetic, so agreement at the chosen smooth point is evidence rather than proof of the complete reverse-mode system. The result prepares the tensor-operation tape used for LLM training in Chapters 15 and 16. Ordinary decoder inference does not run this backward graph: reverse mode is needed while computing training gradients, with fresh pass adjoints kept separate from gradients accumulated across completed backward calls.

The compact Scalar design is an educational example of the general mechanism rather than a claim about the internal representation of a cited Transformer.

Build one fresh reverse pass

Scalar constructors distinguish tracked variables from untracked constants. Both require finite values. Private nodes keep immutable parent-only links, so operations cannot create a cycle. add, mul, neg, sub, exp, and tanh compute one finite primal result and record the ordered parent edges with the local derivative values needed later.

Create finite scalar nodes and record checked results with ordered local derivative edges rust/crates/llm-from-scratch/src/autograd/scalar.rs#scalar-dag-operations

impl Scalar {
    /// Creates a finite leaf whose gradient is tracked.
    pub fn variable(value: f64) -> Result<Self, ScalarAutodiffError> {
        Self::leaf(value, ScalarOperation::Variable, true)
    }

    /// Creates a finite leaf treated as a constant by backpropagation.
    pub fn constant(value: f64) -> Result<Self, ScalarAutodiffError> {
        Self::leaf(value, ScalarOperation::Constant, false)
    }

    fn leaf(
        value: f64,
        operation: ScalarOperation,
        tracked: bool,
    ) -> Result<Self, ScalarAutodiffError> {
        if !value.is_finite() {
            return Err(ScalarAutodiffError::NonFiniteLeaf { operation, value });
        }
        Ok(Self::new_node(value, operation, Vec::new(), tracked))
    }

    fn new_node(
        value: f64,
        operation: ScalarOperation,
        parents: Vec<ParentEdge>,
        tracked: bool,
    ) -> Self {
        Self {
            node: Rc::new(RefCell::new(Node {
                value,
                operation,
                parents,
                gradient: tracked.then_some(0.0),
            })),
        }
    }

    fn operation_node(
        value: f64,
        operation: ScalarOperation,
        parents: Vec<ParentEdge>,
    ) -> Result<Self, ScalarAutodiffError> {
        if !value.is_finite() {
            return Err(ScalarAutodiffError::NonFiniteResult { operation, value });
        }
        debug_assert!(parents.iter().all(|edge| edge.local_derivative.is_finite()));
        let tracked = parents.iter().any(|edge| edge.parent.tracks_gradient());
        Ok(Self::new_node(value, operation, parents, tracked))
    }

    pub fn value(&self) -> f64 {
        self.node.borrow().value
    }

    pub fn operation(&self) -> ScalarOperation {
        self.node.borrow().operation
    }

    pub fn tracks_gradient(&self) -> bool {
        self.node.borrow().gradient.is_some()
    }

    pub fn gradient(&self) -> Option<f64> {
        self.node.borrow().gradient
    }

    /// Returns whether two handles refer to the same graph node.
    pub fn is_same_node(&self, other: &Self) -> bool {
        Rc::ptr_eq(&self.node, &other.node)
    }

    /// Adds two finite scalars and records both ordered operand edges.
    pub fn add(&self, other: &Self) -> Result<Self, ScalarAutodiffError> {
        Self::operation_node(
            self.value() + other.value(),
            ScalarOperation::Add,
            vec![
                ParentEdge {
                    parent: self.clone(),
                    local_derivative: 1.0,
                },
                ParentEdge {
                    parent: other.clone(),
                    local_derivative: 1.0,
                },
            ],
        )
    }

    /// Multiplies two finite scalars and records one edge per operand use.
    pub fn mul(&self, other: &Self) -> Result<Self, ScalarAutodiffError> {
        let left = self.value();
        let right = other.value();
        Self::operation_node(
            left * right,
            ScalarOperation::Multiply,
            vec![
                ParentEdge {
                    parent: self.clone(),
                    local_derivative: right,
                },
                ParentEdge {
                    parent: other.clone(),
                    local_derivative: left,
                },
            ],
        )
    }

    pub fn neg(&self) -> Result<Self, ScalarAutodiffError> {
        Self::operation_node(
            -self.value(),
            ScalarOperation::Negate,
            vec![ParentEdge {
                parent: self.clone(),
                local_derivative: -1.0,
            }],
        )
    }

    pub fn sub(&self, other: &Self) -> Result<Self, ScalarAutodiffError> {
        Self::operation_node(
            self.value() - other.value(),
            ScalarOperation::Subtract,
            vec![
                ParentEdge {
                    parent: self.clone(),
                    local_derivative: 1.0,
                },
                ParentEdge {
                    parent: other.clone(),
                    local_derivative: -1.0,
                },
            ],
        )
    }

    pub fn exp(&self) -> Result<Self, ScalarAutodiffError> {
        let value = self.value().exp();
        Self::operation_node(
            value,
            ScalarOperation::Exp,
            vec![ParentEdge {
                parent: self.clone(),
                local_derivative: value,
            }],
        )
    }

    pub fn tanh(&self) -> Result<Self, ScalarAutodiffError> {
        let value = self.value().tanh();
        Self::operation_node(
            value,
            ScalarOperation::Tanh,
            vec![ParentEdge {
                parent: self.clone(),
                local_derivative: 1.0 - value * value,
            }],
        )
    }

    /// Copies the primal into a new untracked constant with no parent edge.
    pub fn detach(&self) -> Self {
        Self::new_node(self.value(), ScalarOperation::Detached, Vec::new(), false)
    }
}

Backward never starts from stale intermediate gradients. It builds the node-unique topology, seeds a fresh pass-local map at loss, and adds each operand-edge contribution while traversing the topology in reverse. Only after the whole pass and every prospective stored sum are finite does it commit any gradient mutation.

That separation defines repeated-call behavior. The first call commits $\bar{\mathrm{loss}}=1$ , $\bar{\mathrm{square}}=2$ , and $\bar{x}=8$ . The second call recomputes those same pass-local values from scratch, then adds them to storage, giving $2$ , $4$ , and $16$ . It does not send the first call’s stored $\bar{\mathrm{square}}=2$ backward again.

Accumulate every repeated edge in one fresh pass before committing stored gradients rust/crates/llm-from-scratch/src/autograd/scalar.rs#scalar-reverse-pass

    /// Accumulates one fresh reverse pass seeded by one.
    pub fn backward(&self) -> Result<BackwardPass, ScalarAutodiffError> {
        self.backward_with_seed(1.0)
    }

    /// Accumulates one fresh reverse pass without reading stale intermediate grads.
    ///
    /// No stored gradient changes unless every contribution, pass adjoint, and
    /// prospective accumulated gradient is finite.
    pub fn backward_with_seed(&self, seed: f64) -> Result<BackwardPass, ScalarAutodiffError> {
        if !self.tracks_gradient() {
            return Err(ScalarAutodiffError::UntrackedOutput {
                operation: self.operation(),
            });
        }
        if !seed.is_finite() {
            return Err(ScalarAutodiffError::NonFiniteSeed { seed });
        }

        let topology = self.topology();
        let indices = topology
            .iter()
            .enumerate()
            .map(|(index, scalar)| (scalar.key(), index))
            .collect::<HashMap<_, _>>();
        let mut pass_adjoints = vec![0.0; topology.len()];
        pass_adjoints[topology.len() - 1] = seed;
        let mut edges = Vec::new();

        for child in (0..topology.len()).rev() {
            let upstream = pass_adjoints[child];
            let parents = topology[child].node.borrow().parents.clone();
            for (operand, edge) in parents.iter().enumerate() {
                let parent = indices[&edge.parent.key()];
                let contribution = upstream * edge.local_derivative;
                if !contribution.is_finite() {
                    return Err(ScalarAutodiffError::NonFiniteContribution {
                        child,
                        parent,
                        operand,
                        upstream,
                        local_derivative: edge.local_derivative,
                    });
                }

                let parent_tracked = edge.parent.tracks_gradient();
                let (before, after) = if parent_tracked {
                    let previous = pass_adjoints[parent];
                    let next = previous + contribution;
                    if !next.is_finite() {
                        return Err(ScalarAutodiffError::NonFinitePassAdjoint {
                            node: parent,
                            previous,
                            contribution,
                        });
                    }
                    pass_adjoints[parent] = next;
                    (Some(previous), Some(next))
                } else {
                    (None, None)
                };

                edges.push(BackwardEdge {
                    reverse_index: edges.len(),
                    child,
                    parent,
                    operand,
                    local_derivative: edge.local_derivative,
                    upstream,
                    contribution,
                    parent_tracked,
                    parent_adjoint_before: before,
                    parent_adjoint_after: after,
                });
            }
        }

        let prospective = topology
            .iter()
            .enumerate()
            .map(|(index, scalar)| {
                scalar.gradient().map(|stored| {
                    let pass_adjoint = pass_adjoints[index];
                    let accumulated = stored + pass_adjoint;
                    if !accumulated.is_finite() {
                        Err(ScalarAutodiffError::NonFiniteAccumulatedGradient {
                            node: index,
                            stored,
                            pass_adjoint,
                        })
                    } else {
                        Ok(accumulated)
                    }
                })
            })
            .map(|candidate| candidate.transpose())
            .collect::<Result<Vec<_>, _>>()?;

        for (scalar, &gradient) in topology.iter().zip(&prospective) {
            if let Some(gradient) = gradient {
                scalar.node.borrow_mut().gradient = Some(gradient);
            }
        }

        let nodes = topology
            .iter()
            .enumerate()
            .map(|(topology_index, scalar)| BackwardNode {
                topology_index,
                operation: scalar.operation(),
                value: scalar.value(),
                tracked: scalar.tracks_gradient(),
                pass_adjoint: scalar
                    .tracks_gradient()
                    .then_some(pass_adjoints[topology_index]),
                accumulated_gradient: prospective[topology_index],
            })
            .collect();

        Ok(BackwardPass { seed, nodes, edges })
    }

    /// Clears every reachable tracked node without changing the graph or values.
    pub fn zero_grad(&self) {
        for scalar in self.topology() {
            let mut node = scalar.node.borrow_mut();
            if node.gradient.is_some() {
                node.gradient = Some(0.0);
            }
        }
    }

zero_grad clears every reachable tracked node. One new backward call then restores the original first-pass gradients. detach instead creates an untracked constant leaf with the same primal value and no parent. For $x^2+3\operatorname{detach}(x)$ at $x=2$ , the forward value is $10$ , but only $x^2$ reaches the original variable, so its gradient is $4$ .

Typed errors distinguish a constant output, invalid seed, non-finite forward value, local contribution, pass adjoint, and prospective stored sum. Because backward validates before commit, every failed call leaves the stored gradient bits unchanged.

Reject unsafe graph values and reverse contributions without partial mutation rust/crates/llm-from-scratch/src/autograd/scalar.rs#scalar-autodiff-errors

/// The operation that produced a scalar graph node.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScalarOperation {
    Variable,
    Constant,
    Detached,
    Add,
    Multiply,
    Negate,
    Subtract,
    Exp,
    Tanh,
}

impl ScalarOperation {
    /// A stable, locale-neutral name suitable for deterministic evidence.
    pub const fn as_str(self) -> &'static str {
        match self {
            Self::Variable => "variable",
            Self::Constant => "constant",
            Self::Detached => "detached",
            Self::Add => "add",
            Self::Multiply => "mul",
            Self::Negate => "neg",
            Self::Subtract => "sub",
            Self::Exp => "exp",
            Self::Tanh => "tanh",
        }
    }
}

impl fmt::Display for ScalarOperation {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        formatter.write_str(self.as_str())
    }
}

/// A deterministic rejection from scalar graph construction or backpropagation.
#[derive(Clone, Debug, PartialEq)]
pub enum ScalarAutodiffError {
    NonFiniteLeaf {
        operation: ScalarOperation,
        value: f64,
    },
    NonFiniteResult {
        operation: ScalarOperation,
        value: f64,
    },
    UntrackedOutput {
        operation: ScalarOperation,
    },
    NonFiniteSeed {
        seed: f64,
    },
    NonFiniteContribution {
        child: usize,
        parent: usize,
        operand: usize,
        upstream: f64,
        local_derivative: f64,
    },
    NonFinitePassAdjoint {
        node: usize,
        previous: f64,
        contribution: f64,
    },
    NonFiniteAccumulatedGradient {
        node: usize,
        stored: f64,
        pass_adjoint: f64,
    },
}

impl fmt::Display for ScalarAutodiffError {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::NonFiniteLeaf { operation, value } => {
                write!(
                    formatter,
                    "{operation} scalar value {value:?} must be finite"
                )
            }
            Self::NonFiniteResult { operation, value } => {
                write!(
                    formatter,
                    "{operation} produced non-finite scalar value {value:?}"
                )
            }
            Self::UntrackedOutput { operation } => write!(
                formatter,
                "cannot backpropagate from untracked {operation} output"
            ),
            Self::NonFiniteSeed { seed } => {
                write!(formatter, "backward seed {seed:?} must be finite")
            }
            Self::NonFiniteContribution {
                child,
                parent,
                operand,
                upstream,
                local_derivative,
            } => write!(
                formatter,
                "edge {operand} from topology node {child} to {parent} produced a non-finite contribution from upstream {upstream:?} and local derivative {local_derivative:?}"
            ),
            Self::NonFinitePassAdjoint {
                node,
                previous,
                contribution,
            } => write!(
                formatter,
                "topology node {node} cannot accumulate pass adjoint {previous:?} plus contribution {contribution:?}"
            ),
            Self::NonFiniteAccumulatedGradient {
                node,
                stored,
                pass_adjoint,
            } => write!(
                formatter,
                "topology node {node} cannot accumulate stored gradient {stored:?} plus pass adjoint {pass_adjoint:?}"
            ),
        }
    }
}

impl Error for ScalarAutodiffError {}

Chapter 13 supplies a materially separate sampled numerical cross-check. It evaluates the same mathematical function and therefore still shares the function specification, input value, and IEEE f64 arithmetic. Agreement at this selected smooth point is evidence, not proof of the complete reverse graph. For this graph the forward expression is $2x^2$ , so the analytic derivative is $4x$ , which equals $8$ at $x=2$ . The central-difference checker perturbs only the ordinary scalar function; it does not inspect or reuse the backward graph.

Compare reverse mode with a sampled central-difference cross-check and exercise detach rust/demos/ch14-scalar-autodiff/src/lib.rs#nonlinear-detach-gradcheck

/// Keeps the live `x*x` path but stops `detach(x)*3` from reaching `x`.
pub fn detach_example() -> Result<DetachExample, ScalarAutodiffError> {
    let x = Scalar::variable(REUSED_INPUT)?;
    let square = x.mul(&x)?;
    let detached = x.detach();
    let three = Scalar::constant(3.0)?;
    let stopped = detached.mul(&three)?;
    let loss = square.add(&stopped)?;
    loss.backward()?;

    Ok(DetachExample {
        input: x.value(),
        value: loss.value(),
        x_gradient: x.gradient().expect("x is tracked"),
        detached_gradient: detached.gradient(),
    })
}

/// Differentiates a two-operation elementary-function chain.
pub fn nonlinear_example() -> Result<NonlinearExample, ScalarAutodiffError> {
    let x = Scalar::variable(0.5)?;
    let output = x.tanh()?.exp()?;
    output.backward()?;
    Ok(NonlinearExample {
        input: x.value(),
        value: output.value(),
        gradient: x.gradient().expect("x is tracked"),
    })
}

/// Checks the reverse derivative of `2x^2` with Chapter 13 central differences.
pub fn gradcheck_example() -> Result<ScalarGradientCheck, Box<dyn Error>> {
    let x = Scalar::variable(REUSED_INPUT)?;
    let square = x.mul(&x)?;
    let loss = square.add(&square)?;
    loss.backward()?;
    Ok(scalar_gradient_check(
        REUSED_INPUT,
        x.gradient().expect("x is tracked"),
        GRADCHECK_STEP,
        GRADCHECK_TOLERANCE,
        |value| 2.0 * value * value,
    )?)
}

The example reports the shared graph, two successful passes, zeroing, one fresh restoration, detach, and numerical agreement:

Prepare the deterministic scalar autodiff evidence before printing rust/demos/ch14-scalar-autodiff/src/main.rs#learner-scalar-autodiff-output

    let reused = reused_square_example()?;
    let detached = detach_example()?;
    let nonlinear = nonlinear_example()?;
    let gradcheck = gradcheck_example()?;
    let errors = typed_error_example()?;

./course run cargo run --quiet --locked -p ch14-scalar-autodiff

reused square: x=2.000000000000 square=4.000000000000 loss=8.000000000000
one backward: x_grad=8.000000000000 square_grad=2.000000000000 loss_grad=1.000000000000
repeated backward: x_grad=16.000000000000 square_grad=4.000000000000 loss_grad=2.000000000000
zero_grad: x_grad=0.000000000000 square_grad=0.000000000000 loss_grad=0.000000000000
after zero: x_grad=8.000000000000 square_grad=2.000000000000 loss_grad=1.000000000000
detach: expression=x*x+detach(x)*3 value=10.000000000000 x_grad=4.000000000000 detached_grad=none
nonlinear: expression=exp(tanh(x)) input=0.500000000000 value=1.587431271430 gradient=1.248431724655
gradcheck: expression=2*x*x analytic=8.000000000000 numerical=8.000000000000 scaled_error=0.000000000000e0 pass=true
typed errors: constant-output | non-finite-seed | non-finite-accumulated-gradient; gradients unchanged=true
chapter 15 handoff: replace scalar edges with tensor vector-Jacobian products

Follow every operand edge backward

The first diagram keeps the forward and reverse graph invariants visible at once. There are three node cards, not seven copied values. Four individually labeled operand edges still enter the reverse ledger. Each row records reverse order, operand identity, upstream adjoint, local derivative, and contribution. The accumulated node and pass records show the sums, so the two paths cannot collapse into one by color or geometry.

The second diagram records the first and second stored states, zeroing, fresh restoration, the detached branch, the central-difference verdict, and rejected requests. These observations belong beside the graph because together they distinguish a fresh pass-local adjoint from a stored gradient accumulated across completed calls.

Follow every repeated operand edge back to one scalar

Inspect three unique forward nodes, four repeated operand edges, and every ordered contribution in one fresh reverse pass.

Scalar loss: 8.000000000000
Unique graph nodes: 3
Operand edges: 4

Build one shared forward graph

Node identity removes duplicate visits from the topology; it never removes operand edges. Both x times x and square plus square retain two ordered uses.

x

Forward order

0

Operation

tracked variable

Primal value

2.000000000000

Adjoint in this pass

8.000000000000
square

Forward order

1

Operation

multiplication

Primal value

4.000000000000

Adjoint in this pass

2.000000000000
- Operand use 0 x
- Operand use 1 x
loss

Forward order

2

Operation

addition

Primal value

8.000000000000

Adjoint in this pass

1.000000000000
- Operand use 0 square
- Operand use 1 square

Accumulate one fresh reverse pass

The output seed reaches square twice. Square waits for both contributions, then sends its accumulated adjoint through both multiplication operands.

Reverse order	Consuming result	Operand use	Operand value	Local derivative	Upstream adjoint	Edge contribution
0	loss	0	square	1.000000000000	1.000000000000	1.000000000000
1	loss	1	square	1.000000000000	1.000000000000	1.000000000000
2	square	0	x	2.000000000000	2.000000000000	4.000000000000
3	square	1	x	2.000000000000	2.000000000000	4.000000000000

Separate a fresh reverse pass from stored gradient state

Compare repeated passes, zeroing, detach, numerical agreement, and rejected requests without mixing pass-local adjoints with stored gradients.

Commit, repeat, zero, and restore

Each call computes fresh pass-local adjoints before adding one complete pass to storage. It never propagates the previous call's intermediates.

first pass committed

x: 8.000000000000
square: 2.000000000000
loss: 1.000000000000

second pass accumulated

x: 16.000000000000
square: 4.000000000000
loss: 2.000000000000

stored gradients zeroed

x: 0.000000000000
square: 0.000000000000
loss: 0.000000000000

one fresh pass restored

x: 8.000000000000
square: 2.000000000000
loss: 1.000000000000

Check detach and a sampled numerical cross-check

Detach preserves a primal value but removes the parent edge. Chapter 13 evaluates the same mathematical function through a separate sampled central-difference path; because both paths share the function specification, input, and f64 arithmetic, agreement is evidence rather than proof.

detached branch stopped

Expression: x*x+detach(x)*3
Primal value: 10.000000000000
Stored gradient: 4.000000000000

nonlinear chain differentiated

Expression: exp(tanh(x))
Primal value: 1.587431271430
Stored gradient: 1.248431724655

numerical check passed

Analytic gradient: 8.000000000000
Numerical gradient: 8.000000000000
Scaled error: 0.000000000000e0
Tolerance: 1.000000000000e-9

Reject unsafe gradients before mutation

The implementation validates every contribution and prospective stored sum before committing, so a failed backward call leaves all stored gradient bits unchanged.

constant output has no tracked gradient path

Operation: constant

backward seed is not finite

Backward seed: inf

prospective stored gradient is not finite

Graph node: 0

Predict before running Rust

Predict square and loss before inspecting the graph output.
Count unique nodes and operand edges separately. Why are the counts different?
Compute both contributions to $\bar{\mathrm{square}}$ and both to $\bar{x}$ .
Predict the gradients if each parent used assignment instead of addition.
Predict stored gradients after two backward calls, after zeroing, and after one new call.
Predict the value and original $x$ gradient of $x^2+3\operatorname{detach}(x)$ at $x=2$ .
Explain why a failed final stored-gradient check must not commit earlier valid nodes.
Misconception check: does reverse mode approximate derivatives, select one branch, or run in ordinary decoder inference?

Check the eight scalar-autodiff predictions

$\mathrm{square}=2\cdot2=4$ ; $\mathrm{loss}=4+4=8$ .
There are three node identities and four parent edges. Each operation result is visited once, but each repeated operand remains a separate derivative path.
Addition contributes $1+1$ , so $\bar{\mathrm{square}}=2$ . Multiplication contributes $2\cdot2+2\cdot2$ , so $\bar{x}=8$ .
Assignment would retain one of the two equal addition contributions, giving $\bar{\mathrm{square}}=1$ , and then one of the two equal multiplication contributions, giving $\bar{x}=2$ . This symmetric graph produces the same wrong values in either edge order. In a nonsymmetric graph, which contribution survives would also depend on traversal order.
Stored $(\bar{\mathrm{loss}},\bar{\mathrm{square}},\bar{x})$ is $(1,2,8)$ , then $(2,4,16)$ , then $(0,0,0)$ , then $(1,2,8)$ . Every backward call computes a fresh pass before storage accumulation.
The value is $4+6=10$ . The detached branch has no edge to the original $x$ , so its gradient is only $2x=4$ .
Partial mutation would make a failed pass depend on how far traversal got. Validating every pass value and prospective stored sum first preserves all earlier bits on error.
None. Reverse mode applies exact local derivative rules at the stored primal values and adds every graph path. This graph supports training; ordinary generation uses only the forward decoder.

Run the example after predicting:

./course run cargo run --quiet --locked -p ch14-scalar-autodiff

Prepare tensor reverse mode

The cumulative project can now record scalar dependencies, traverse each reachable node once in reverse topological order, add every operand-edge contribution, safely accumulate complete fresh passes, zero gradients, detach a value, and verify analytic results numerically. Chapter 15 replaces scalar nodes with tensor-operation VJPs for reshape, transpose, broadcasts, and reductions while preserving these reverse-accumulation rules.

One graph node per scalar is deliberately a teaching model, not a practical way to represent every element of an LLM activation. The next chapter records one node per tensor operation and carries shape context backward. Its VJPs must still honor exactly what this example exposed: unique topological visits, multiple operand contributions, fresh pass-local state, deliberate stored accumulation, explicit zeroing, and disconnected detached values.