Chapter 06 · Content revision 4

Initialize residual projection weights for decoder depth

Scale each block's two output projection weight matrices once at initialization, then inspect residual activation RMS and completed mean-loss adjoints on fixed CPU decoders.

The problem: depth adds more residual contributions

Each complete decoder block adds an attention output to the residual state, then adds a feed-forward output. Increasing the block count repeats these additions. The existing Xavier initializer uses a matrix’s input and output widths to choose its scale; those widths can stay fixed while the number of residual additions grows. A suitable scale for one matrix therefore does not settle how activations and loss sensitivity behave throughout the stack.

We will reduce the initial coefficients of the two projections that write branch outputs into the residual state. This is a local initialization heuristic. We will also observe the resulting decoder before training, using the same occurrence-local contexts and valid-target mean loss as Chapter 5.

Start with four selected coefficients

Take an illustrative output-projection vector $[0.2,-0.4,0,0.1]$ . For one complete block, keep it unchanged. For two blocks, multiply each coefficient once by $1/\sqrt2\approx0.707107$ . In their original order, the stored coefficients are approximately $0.141421$ , $-0.282843$ , $0$ and $0.070711$ . For eight blocks, the factor is $1/\sqrt8\approx0.353553$ , giving approximately $0.070711$ , $-0.141421$ , $0$ and $0.035355$ .

These decimal displays are rounded to six places; the Rust demo emits the actual f64 results. The vector illustrates the transformation, rather than estimating a distribution from four values. An unselected query-projection vector $[0.2,-0.4,0,0.1]$ remains unchanged at all three depths.

The selected matrices are blocks.{b}.attention.output.weight and blocks.{b}.ffn.down.weight, where {b} is the zero-based block index. The attention output projection writes the combined attention features back to model width; the feed-forward down projection writes the SwiGLU features back to that width. Query, key and value projections, gate and up projections, embeddings and their tied vocabulary head, and all RMSNorm gains retain their base values.

At fixed branch inputs, each selected projection is linear in its weights, so multiplying those weights by the factor multiplies that branch’s output by the factor. In a whole decoder, later branch inputs also change. The final decoder output is therefore not simply the unscaled output times one factor.

Apply the factor once, using complete blocks

Use a positive number of complete decoder blocks, $L>0$ , for the initialization rule:

\alpha=L^{-1/2},\qquad W_{\mathrm{stored}}=\alpha W_{\mathrm{base}}

Apply this rule to each of the two selected matrices in every block. A block has two residual additions, but the depth divisor remains the complete-block count $L$ . Eight blocks have $16$ additions and use $1/\sqrt8$ , rather than $1/\sqrt{16}$ .

The transformation happens before forward execution. Forward uses $W_{\mathrm{stored}}$ directly and adds no further factor. After initialization, these stored coefficients may evolve normally when a later chapter trains the model; they are not permanently constrained to remain a scaled base vector.

One block has factor $1$ . The implementation returns its base decoder directly, bypassing both transformation and reconstruction. It preserves parameter bits, names, shapes and ordering, random-number consumption, forward values, loss and parameter adjoints relative to the existing one-block constructor. The old constructor still admits zero-block models; this new policy requires positive depth because its factor is undefined at zero.

What the variance intuition assumes

Consider a toy sum of $2L$ independent, zero-mean increments, each with variance $\sigma^2$ . Multiplying each increment by $\alpha$ gives

\operatorname{Var}\left(\sum_{k=1}^{2L}\alpha r_k\right) =2L\alpha^2\sigma^2=2\sigma^2.

Here $r_k$ is one toy increment. Independence removes covariance terms, and equal variance gives the same contribution for each increment. This calculation motivates the depth factor. Real attention and feed-forward branches depend on shared residual states, and their Jacobians can be correlated. They do not satisfy this toy model in general, so the calculation is no proof of stable activations, gradients or training for the decoder.

Differentiate the coefficient actually stored

For a scalar product with input $2$ , write $y=2w_{\mathrm{stored}}$ . Its derivative is

\frac{\partial y}{\partial w_{\mathrm{stored}}}=2.

If instead we keep a base coordinate in the graph and define $w_{\mathrm{stored}}=\alpha w_{\mathrm{base}}$ , the chain rule gives

\frac{\partial y}{\partial w_{\mathrm{base}}}=2\alpha.

Our decoder stores the initialized coefficient as the parameter. Its gradient is with respect to that stored coefficient, so it receives no second automatic depth factor. The two coordinate choices do not imply equal parameter updates under an optimizer, nor do these derivatives imply convergence.

What the initialization symbols mean

$L$ counts complete decoder blocks and must be positive. Each block contains an attention addition followed by a feed-forward addition.
$\alpha$ is the dimensionless initialization factor $1/\sqrt L$ .
$W_{\mathrm{base}}$ is one selected output-projection matrix produced by the existing Xavier initializer before this transformation.
$W_{\mathrm{stored}}$ is that matrix after initialization. Forward and parameter differentiation use its stored values directly.

Construct fresh parameters and observe the existing decoder

DepthScalePolicy::new requires a positive complete-block count and computes the factor. Role selection is applied only after the existing decoder validator has checked the complete parameter names, order and shapes. This prevents a plausible suffix match from standing in for a valid decoder layout.

Compute the positive complete-block factor and select weights in the attention output and feed-forward down projections. rust/crates/llm-from-scratch-practical/src/nn/residual_scale.rs#depth-initialization-factor

/// A checked factor for a positive number of complete decoder blocks.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct DepthScalePolicy {
    depth: usize,
    factor: f64,
}

impl DepthScalePolicy {
    pub fn new(depth: usize) -> Result<Self, DepthScaleError> {
        if depth == 0 {
            return Err(DepthScaleError::ZeroDepth);
        }
        let factor = 1.0 / (depth as f64).sqrt();
        if !factor.is_finite() || factor <= 0.0 {
            return Err(DepthScaleError::InvalidFactor { depth, factor });
        }
        Ok(Self { depth, factor })
    }

    pub const fn depth(self) -> usize {
        self.depth
    }

    pub const fn factor(self) -> f64 {
        self.factor
    }

    /// Called only after the decoder owner validates the entire parameter layout.
    pub(crate) fn selects(self, name: &str) -> bool {
        name.ends_with(".attention.output.weight") || name.ends_with(".ffn.down.weight")
    }
}

FreshDepthInitialization::prepare checks the configuration’s parameter count and position capacity against the caller’s DepthInitializationBudget before allocating model arrays. It clones the caller’s SplitMix64 state and calls the existing DecoderModel::new once using that clone. Dropping the preparation leaves the caller’s generator unchanged.

complete first refuses parameter updates or nonzero accumulated gradients. This refusal precedes any weight transformation or RNG commit. Zero gradients from construction, or from zero_grad when parameters have not been updated, are allowed.

complete consumes this fresh state. For depths greater than one, it stages the selected stored values and rebuilds through DecoderModel::from_parameters. Only after successful assembly does it commit the trial random-generator state to the caller. Refusal leaves the caller’s state unchanged. No separate decoder, Xavier generator or forward multiplier is needed.

Complete fresh initialization of decoder projection weights, bypass rebuilding at one block, and commit RNG after successful assembly. rust/crates/llm-from-scratch-practical/src/nn/depth_init.rs#selective-fresh-depth-initialization

    pub fn complete(self) -> Result<DepthInitializedDecoder, DepthInitError> {
        for (parameter, &version) in self.base.parameters().iter().zip(&self.base_versions) {
            if parameter.tensor().value_revision() != version
                || parameter
                    .tensor()
                    .gradient()
                    .is_some_and(|gradient| gradient.as_slice().iter().any(|&x| x != 0.0))
            {
                return Err(DepthInitError::ModifiedFreshParameter {
                    name: parameter.name().to_owned(),
                });
            }
        }
        let config = self.base.config();
        let transformed_tensors = if self.policy.depth() == 1 {
            0
        } else {
            2 * self.policy.depth()
        };
        let model = if self.policy.depth() == 1 {
            // Preserve exact leaves, values and RNG consumption for the scalar case.
            self.base
        } else {
            let staged = scaled_parameters(config, self.base.parameters(), self.policy)?;
            DecoderModel::from_parameters(config, staged)?
        };
        let record = DepthInitializationRecord {
            config,
            factor: self.policy.factor(),
            initial_rng_state: self.initial_rng_state,
            final_rng_state: self.trial_rng.state(),
            transformed_tensors,
        };
        *self.caller_rng = self.trial_rng;
        Ok(DepthInitializedDecoder { model, record })
    }

The stored parameters continue through the existing block’s RMSNorm, attention, SwiGLU and two residual additions.

Use stored parameters in the existing normalized attention and SwiGLU branches and their two residual additions. rust/crates/llm-from-scratch-practical/src/models/decoder_block.rs#decoder-block-layer

/// Every inspectable value produced by the two pre-normalized residual paths.
#[derive(Clone, Debug)]
pub struct DecoderBlockForward {
    attention_norm: RmsNormForward,
    attention: MultiHeadAttentionForward,
    after_attention: TensorValue,
    feed_forward_norm: RmsNormForward,
    feed_forward: SwiGluForward,
    output: TensorValue,
}

impl DecoderBlockForward {
    pub fn attention_norm(&self) -> &RmsNormForward {
        &self.attention_norm
    }

    pub fn attention(&self) -> &MultiHeadAttentionForward {
        &self.attention
    }

    pub fn attention_weights(&self) -> &TensorValue {
        self.attention.attention_weights()
    }

    pub fn after_attention(&self) -> &TensorValue {
        &self.after_attention
    }

    pub fn feed_forward_norm(&self) -> &RmsNormForward {
        &self.feed_forward_norm
    }

    pub fn feed_forward(&self) -> &SwiGluForward {
        &self.feed_forward
    }

    pub fn output(&self) -> &TensorValue {
        &self.output
    }

    pub fn into_output(self) -> TensorValue {
        self.output
    }
}

/// RMSNorm → causal MHA → residual, then RMSNorm → SwiGLU → residual.
#[derive(Clone, Debug)]
pub struct DecoderBlock {
    attention_norm: RmsNorm,
    attention: MultiHeadAttention,
    feed_forward_norm: RmsNorm,
    feed_forward: SwiGlu,
    parameters: NamedParameters,
    model_width: usize,
}

impl DecoderBlock {
    /// Initializes every matrix transactionally from one deterministic stream.
    pub fn new(
        parameter_prefix: impl Into<String>,
        config: DecoderBlockConfig,
        rng: &mut SplitMix64,
    ) -> Result<Self, DecoderBlockError> {
        let parameter_prefix = parameter_prefix.into();
        let attention_norm = RmsNorm::new(
            format!("{parameter_prefix}.attention_norm.gain"),
            config.model_width,
            config.rms_epsilon,
        )
        .map_err(DecoderBlockError::AttentionNorm)?;
        let feed_forward_norm = RmsNorm::new(
            format!("{parameter_prefix}.ffn_norm.gain"),
            config.model_width,
            config.rms_epsilon,
        )
        .map_err(DecoderBlockError::FeedForwardNorm)?;

        let mut trial = rng.clone();
        let attention = MultiHeadAttention::new_with_attention(
            format!("{parameter_prefix}.attention"),
            config.model_width,
            config.heads,
            config.kv_heads,
            config.max_positions,
            config.rope_base,
            config.context_window,
            &mut trial,
        )
        .map_err(DecoderBlockError::Attention)?;
        let feed_forward = SwiGlu::new(
            format!("{parameter_prefix}.ffn"),
            config.model_width,
            config.feed_forward_width,
            config.model_width,
            &mut trial,
        )
        .map_err(DecoderBlockError::FeedForward)?;
        let block = Self::from_parts(attention_norm, attention, feed_forward_norm, feed_forward)?;
        *rng = trial;
        Ok(block)
    }

    /// Assembles already named deterministic components after cross-width checks.
    pub fn from_parts(
        attention_norm: RmsNorm,
        attention: MultiHeadAttention,
        feed_forward_norm: RmsNorm,
        feed_forward: SwiGlu,
    ) -> Result<Self, DecoderBlockError> {
        let model_width = attention.model_width();
        for (component, actual) in [
            (
                DecoderBlockComponent::AttentionNorm,
                attention_norm.feature_width(),
            ),
            (
                DecoderBlockComponent::FeedForwardNorm,
                feed_forward_norm.feature_width(),
            ),
            (
                DecoderBlockComponent::FeedForwardInput,
                feed_forward.input_width(),
            ),
            (
                DecoderBlockComponent::FeedForwardOutput,
                feed_forward.output_width(),
            ),
        ] {
            if actual != model_width {
                return Err(DecoderBlockError::ComponentWidthMismatch {
                    component,
                    expected: model_width,
                    actual,
                });
            }
        }

        let mut listed = Vec::with_capacity(9);
        listed.extend(attention_norm.parameters().iter().cloned());
        listed.extend(attention.parameters().iter().cloned());
        listed.extend(feed_forward_norm.parameters().iter().cloned());
        listed.extend(feed_forward.parameters().iter().cloned());
        let parameters = NamedParameters::try_new(listed)?;

        Ok(Self {
            attention_norm,
            attention,
            feed_forward_norm,
            feed_forward,
            parameters,
            model_width,
        })
    }

    /// Runs the two transformation branches in exact pre-normalized order.
    pub fn forward(
        &self,
        input: &TensorValue,
        position_offset: usize,
    ) -> Result<DecoderBlockForward, DecoderBlockError> {
        self.forward_with_context(AutogradContext::recording(), input, position_offset)
    }

    /// Runs both pre-normalized branches under an explicit recording policy.
    pub fn forward_with_context(
        &self,
        context: AutogradContext,
        input: &TensorValue,
        position_offset: usize,
    ) -> Result<DecoderBlockForward, DecoderBlockError> {
        self.forward_with_optional_lengths(context, input, position_offset, None)
    }

    pub(crate) fn forward_with_optional_lengths(
        &self,
        context: AutogradContext,
        input: &TensorValue,
        position_offset: usize,
        lengths: Option<&[usize]>,
    ) -> Result<DecoderBlockForward, DecoderBlockError> {
        self.forward_with_layout(context, input, position_offset, lengths, None, None)
    }

    pub(crate) fn forward_with_layout(
        &self,
        context: AutogradContext,
        input: &TensorValue,
        position_offset: usize,
        lengths: Option<&[usize]>,
        starts: Option<&[usize]>,
        positions: Option<&[usize]>,
    ) -> Result<DecoderBlockForward, DecoderBlockError> {
        self.forward_with_layout_and_dropout(
            context,
            input,
            position_offset,
            lengths,
            starts,
            positions,
            0,
            &mut DropoutExecution::Identity,
        )
    }

    #[allow(clippy::too_many_arguments)]
    pub(crate) fn forward_with_layout_and_dropout(
        &self,
        context: AutogradContext,
        input: &TensorValue,
        position_offset: usize,
        lengths: Option<&[usize]>,
        starts: Option<&[usize]>,
        positions: Option<&[usize]>,
        layer: usize,
        dropout: &mut DropoutExecution<'_>,
    ) -> Result<DecoderBlockForward, DecoderBlockError> {
        let attention_norm = self
            .attention_norm
            .forward_with_intermediates_and_context(context, input)
            .map_err(DecoderBlockError::AttentionNorm)?;
        let attention = self
            .attention
            .forward_with_layout(
                context,
                attention_norm.output(),
                position_offset,
                lengths,
                starts,
                positions,
            )
            .map_err(DecoderBlockError::Attention)?;
        let (after_attention, feed_forward_norm, feed_forward, output) =
            self.finish_attention_residual(context, input, attention.output(), layer, dropout)?;

        Ok(DecoderBlockForward {
            attention_norm,
            attention,
            after_attention,
            feed_forward_norm,
            feed_forward,
            output,
        })
    }

    #[allow(clippy::too_many_arguments)]
    pub(crate) fn forward_output_with_layout_and_dropout(
        &self,
        context: AutogradContext,
        input: &TensorValue,
        position_offset: usize,
        lengths: Option<&[usize]>,
        starts: Option<&[usize]>,
        positions: Option<&[usize]>,
        layer: usize,
        dropout: &mut DropoutExecution<'_>,
    ) -> Result<TensorValue, DecoderBlockError> {
        if context.execution().attention_tiles().is_none() {
            return self
                .forward_with_layout_and_dropout(
                    context,
                    input,
                    position_offset,
                    lengths,
                    starts,
                    positions,
                    layer,
                    dropout,
                )
                .map(DecoderBlockForward::into_output);
        }
        let normalized = self
            .attention_norm
            .forward_with_context(context, input)
            .map_err(DecoderBlockError::AttentionNorm)?;
        let attended = self
            .attention
            .forward_output_with_layout(
                context,
                &normalized,
                position_offset,
                lengths,
                starts,
                positions,
            )
            .map_err(DecoderBlockError::Attention)?;
        self.finish_attention_residual(context, input, &attended, layer, dropout)
            .map(|(_, _, _, output)| output)
    }

    fn finish_attention_residual(
        &self,
        context: AutogradContext,
        input: &TensorValue,
        attention_output: &TensorValue,
        layer: usize,
        dropout: &mut DropoutExecution<'_>,
    ) -> Result<(TensorValue, RmsNormForward, SwiGluForward, TensorValue), DecoderBlockError> {
        let branch = dropout
            .apply(
                context,
                attention_output,
                DropoutSiteId {
                    layer,
                    kind: DropoutSiteKind::AttentionOutput,
                },
            )
            .map_err(DecoderBlockError::Dropout)?;
        let after_attention = residual_add_with_context(context, input, &branch)
            .map_err(DecoderBlockError::AttentionResidual)?;
        let (feed_forward_norm, feed_forward, output) =
            self.forward_feed_forward_path_and_dropout(context, &after_attention, layer, dropout)?;

        Ok((after_attention, feed_forward_norm, feed_forward, output))
    }

    /// One FFN residual path, reused by full blocks and residual-site diagnostics.
    pub(crate) fn forward_feed_forward_path(
        &self,
        context: AutogradContext,
        after_attention: &TensorValue,
    ) -> Result<(RmsNormForward, SwiGluForward, TensorValue), DecoderBlockError> {
        self.forward_feed_forward_path_and_dropout(
            context,
            after_attention,
            0,
            &mut DropoutExecution::Identity,
        )
    }

    pub(crate) fn forward_feed_forward_path_and_dropout(
        &self,
        context: AutogradContext,
        after_attention: &TensorValue,
        layer: usize,
        dropout: &mut DropoutExecution<'_>,
    ) -> Result<(RmsNormForward, SwiGluForward, TensorValue), DecoderBlockError> {
        let feed_forward_norm = self
            .feed_forward_norm
            .forward_with_intermediates_and_context(context, after_attention)
            .map_err(DecoderBlockError::FeedForwardNorm)?;
        let feed_forward = self
            .feed_forward
            .forward_with_intermediates_and_context(context, feed_forward_norm.output())
            .map_err(DecoderBlockError::FeedForward)?;
        let branch = dropout
            .apply(
                context,
                feed_forward.output(),
                DropoutSiteId {
                    layer,
                    kind: DropoutSiteKind::FeedForwardOutput,
                },
            )
            .map_err(DecoderBlockError::Dropout)?;
        let output = residual_add_with_context(context, after_attention, &branch)
            .map_err(DecoderBlockError::FeedForwardResidual)?;
        Ok((feed_forward_norm, feed_forward, output))
    }

    pub fn attention_norm(&self) -> &RmsNorm {
        &self.attention_norm
    }

    pub fn attention(&self) -> &MultiHeadAttention {
        &self.attention
    }

    pub fn feed_forward_norm(&self) -> &RmsNorm {
        &self.feed_forward_norm
    }

    pub fn feed_forward(&self) -> &SwiGlu {
        &self.feed_forward
    }

    pub fn parameters(&self) -> &[NamedParameter] {
        self.parameters.as_slice()
    }

    pub const fn model_width(&self) -> usize {
        self.model_width
    }

    pub fn parameter_count(&self) -> usize {
        self.parameters
            .as_slice()
            .iter()
            .map(|parameter| parameter.tensor().value().len())
            .sum()
    }
}

initialize_depth_scaled distinguishes Fresh, Initialized and Restored sources. Only Fresh is accepted; the other states are refused before random draws or mutation. Applying initialization again would change coefficients that already carry their initialization or training history.

Restoration uses DepthInitializedDecoder::from_initialized_parameters with the typed construction record and the actual stored parameters. It restores those supplied values exactly and never rescales them. The record identifies the configuration, policy, base initializer and construction RNG states. It does not supply a trained-checkpoint wire format or optimizer state.

Reconstruct the decoder from validated named stored parameters without drawing or scaling weights again. rust/crates/llm-from-scratch-practical/src/models/decoder.rs#decoder-parameter-rebuild

    /// Rebuilds every component handle from one exact stable-order parameter set.
    ///
    /// State restoration uses this construction boundary to create an isolated
    /// decoder. Ordinary optimizer steps instead update the existing leaves, so
    /// the registry, components, and tied embedding keep their live aliases.
    pub fn from_parameters(
        config: DecoderModelConfig,
        parameters: Vec<NamedParameter>,
    ) -> Result<Self, DecoderModelError> {
        validate_parameter_layout(config, parameters.as_slice())?;
        let embedding = Embedding::from_parameter(parameters[0].clone())
            .map_err(DecoderModelError::Embedding)?;
        let (blocks, final_norm) =
            assemble_backbone(config.block_config(), config.layers, &parameters[1..])?;
        Self::from_parts(config, embedding, blocks, final_norm)
    }

Measure residual magnitude and completed loss sensitivity

For residual values $x$ at $n$ valid token positions with $d$ features each, define root mean square as

\operatorname{RMS}(x) =\sqrt{\frac{1}{nd}\sum_{t\in\mathcal V}\sum_{c=1}^{d}x_{tc}^{2}}.

$\mathcal V$ is the set of valid positions and has size $n$ ; $c$ indexes features. RMS measures magnitude about zero. It does not subtract a mean and is not centered variance. RMSNorm also does not center activations or establish unit variance.

Record the input residual as site $0$ , then the state after each attention addition and each feed-forward addition in forward order. Adjacent blocks share a boundary, which is counted once. There are $2L+1$ sites, ending at site $2L$ before final model RMSNorm and the tied vocabulary head.

Let $A_s$ be activation RMS at site $s$ . Let $G_s$ be RMS of the fully accumulated derivative of the existing valid-target mean cross entropy with respect to that site’s residual features. The mean loss uses natural logarithms and is reported in nats per valid target. Its derivatives describe sensitivity of that same objective; they are not parameter-gradient norms or partial contributions from one backward edge.

Use two different normalizers:

a_s=\frac{A_s}{A_0},\qquad g_s=\frac{G_s}{G_{2L}}.

$a_s$ and $g_s$ are dimensionless ratios. Activations are compared with the input magnitude. Adjoint magnitudes are compared with the terminal residual adjoint, before the final RMSNorm and tied head; those later operations still contribute to that terminal derivative. The normalization fixes $a_0=1$ and $g_{2L}=1$ . It does not fix the absolute magnitudes of either normalizer.

Both normalizers must be positive and finite. Zero or nonfinite values are errors; the observer does not add an epsilon to make a ratio reportable. It includes only valid-position features, skipping padding before arithmetic, and uses an overflow-safe RMS reduction. Multiplying a padded NaN by zero would not exclude it safely. Empty validity, nonfinite valid values and missing completed adjoints are also errors.

The observer selects residual handles by exact node identity after the existing reverse pass has accumulated all their contributions. RMS calculations take place outside the differentiation graph. Observation adds no graph operations, random draws or changes to weight values; the usual mean-loss reverse pass still produces the decoder’s parameter gradients.

Request completed adjoints for selected residual handles through the existing tensor reverse pass. rust/crates/llm-from-scratch-practical/src/autograd/tensor_core.rs#completed-selected-adjoints

    /// Uses the existing reverse pass and copies completed adjoints in request order.
    ///
    /// Missing or untracked nodes yield `None`. Node identity, rather than a
    /// matching shape or operation name, binds each observation. Parameter
    /// gradients have exactly the ordinary reverse pass's accumulation semantics.
    pub fn backward_with_selected_adjoints(
        &self,
        selected: &[TensorValue],
    ) -> Result<Vec<Option<Tensor>>, TensorAutodiffError> {
        self.backward_scalar(RecordSelectedAdjoints { selected })
    }

Measure valid residual features and completed mean-loss adjoints, using input and terminal RMS as separate normalizers. rust/crates/llm-from-scratch-practical/src/nn/depth_health.rs#completed-residual-health

pub fn observe_residual_health(
    model: &DecoderModel,
    forward: &DecoderModelForward,
    mean_loss: &TensorValue,
    valid: &[bool],
) -> Result<DepthHealthReport, DepthHealthError> {
    let sites = residual_sites(forward);
    if forward.blocks().len() != model.config().layers()
        || forward.embedding().shape().last().copied() != Some(model.config().model_width())
    {
        return Err(DepthHealthError::ModelForwardMismatch);
    }
    let activations = sites
        .iter()
        .map(|site| valid_feature_rms(&site.value(), valid))
        .collect::<Result<Vec<_>, _>>()?;
    let adjoints = mean_loss.backward_with_selected_adjoints(&sites)?;
    let gradients = adjoints
        .iter()
        .enumerate()
        .map(|(site, adjoint)| {
            let adjoint = adjoint
                .as_ref()
                .ok_or(DepthHealthError::MissingAdjoint { site })?;
            if adjoint.shape() != sites[site].shape() {
                return Err(DepthHealthError::InvalidGeometry);
            }
            valid_feature_rms(adjoint, valid)
        })
        .collect::<Result<Vec<_>, _>>()?;
    let input_activation_normalizer = activations[0];
    let terminal_adjoint_normalizer = *gradients.last().ok_or(DepthHealthError::InvalidGeometry)?;
    if input_activation_normalizer <= 0.0
        || terminal_adjoint_normalizer <= 0.0
        || !input_activation_normalizer.is_finite()
        || !terminal_adjoint_normalizer.is_finite()
    {
        return Err(DepthHealthError::InvalidNormalizer);
    }
    let rows = activations
        .iter()
        .zip(&gradients)
        .enumerate()
        .map(|(site, (&a, &g))| {
            let activation_ratio = a / input_activation_normalizer;
            let adjoint_ratio = g / terminal_adjoint_normalizer;
            let label = if site == 0 {
                "input".to_owned()
            } else if site % 2 == 1 {
                format!("block.{}.after_attention", site / 2)
            } else {
                format!("block.{}.after_feed_forward", site / 2 - 1)
            };
            Ok(ResidualHealthRow {
                site,
                label,
                activation_rms: a,
                adjoint_rms: g,
                activation_ratio,
                adjoint_ratio,
                activation_status: classify_ratio(activation_ratio)?,
                adjoint_status: classify_ratio(adjoint_ratio)?,
            })
        })
        .collect::<Result<Vec<_>, DepthHealthError>>()?;
    let mut branches = Vec::with_capacity(forward.blocks().len());
    for (block, (values, owner)) in forward.blocks().iter().zip(model.blocks()).enumerate() {
        branches.push(BranchHealthRow {
            block,
            attention_output_rms: valid_feature_rms(&values.attention().output().value(), valid)?,
            feed_forward_output_rms: valid_feature_rms(
                &values.feed_forward().output().value(),
                valid,
            )?,
            attention_norm_gain: owner
                .attention_norm()
                .gain()
                .tensor()
                .value()
                .as_slice()
                .to_vec(),
            feed_forward_norm_gain: owner
                .feed_forward_norm()
                .gain()
                .tensor()
                .value()
                .as_slice()
                .to_vec(),
        });
    }
    Ok(DepthHealthReport {
        valid_positions: valid.iter().filter(|&&v| v).count(),
        width: model.config().model_width(),
        input_activation_normalizer,
        terminal_adjoint_normalizer,
        sites: rows,
        branches,
        final_norm_gain: model
            .final_norm()
            .gain()
            .tensor()
            .value()
            .as_slice()
            .to_vec(),
    })
}

Classify each finite, unrounded ratio in the inclusive band $[0.01,100]$ . This course-local band catches large relative changes in either direction. It does not establish convergence or adequate absolute gradients. Branch-output RMS and norm-gain vectors are reported separately; they are outside this residual-site classification.

Run the fixed CPU example

With Docker running and the repository’s Rust workspace setup available, run from the repository root:

COURSE_CORPUS=false ./course run cargo run --offline --locked -p practical-ch06-depth-stable-decoder

The example uses supplied IDs and needs no corpus or model download. Seed 4701 initializes each fixture. Source [0,99,100,101,1] gives inputs [0,99,100,101] and targets [99,100,101,1], with positions [0,1,2,3], all valid in one segment. In layout-v1, 0 is BOS and 1 is EOS. These are controlled valid-ID examples, not a claim about trained tokenizer segmentation or a real corpus. RoPE base is $10000$ and RMSNorm epsilon is $10^{-6}$ .

The JSON output contains worked_values, four fixtures, synthetic_classifiers and a symbolic laptop_plan. Each fixture reports initialized and same_base_unscaled_control, their residual sites and within_band: true. It measures before any optimizer update.

Print initialized projection coefficients, measured residual activations and adjoints, synthetic ratios and symbolic laptop metadata. rust/demos/practical-ch06-depth-stable-decoder/src/main.rs#depth-stable-example

fn main() -> Result<(), Box<dyn std::error::Error>> {
    println!(
        "{}",
        serde_json::to_string_pretty(&practical_ch06_depth_stable_decoder::demo_trace()?)?
    );
    Ok(())
}

History: fan-based scale and depth-aware residual methods

Fan-in and fan-out reasoning sets a matrix’s initialization scale under simplified assumptions; it does not by itself account for repeated additions in this residual decoder.

Glorot and Bengio’s 2010 paper studies activation and backward-gradient variance near initialization.

Section 4.2 derives fan-in and fan-out variance conditions in an approximately linear regime with independently initialized weights and equal input-feature variances, and gives a compromise normalized initialization.

Fan-in counts the inputs to a matrix and fan-out counts its outputs. Their compromise initialization targets ideal weight variance $2/(\mathrm{fan\_in}+\mathrm{fan\_out})$ .

For a $4\times4$ matrix, the existing Xavier uniform initializer therefore has ideal half-bound $\sqrt{3/4}$ and variance $1/4$ . Multiplying a selected matrix by $\alpha$ multiplies its half-bound by $\alpha$ and its ideal variance by $\alpha^2$ .

Ideal uniform distribution for a selected four-by-four matrix after initialization
Complete blocks	Uniform half-bound	Weight variance
$1$	$\sqrt{3/4}$	$1/4$
$2$	$\sqrt{3/8}$	$1/8$
$8$	$\sqrt{3/32}$	$1/32$

These are ideal distribution statements. The empirical variance of a finite matrix need not equal them, and the four illustrative coefficients earlier are not an empirical check of these variances. The paper’s activation setting and simplified assumptions also differ from our residual RMSNorm/SwiGLU stack.

Compute the fan-based Xavier scale and sample a fresh uniform matrix with the existing initializer. rust/crates/llm-from-scratch-practical/src/nn/init.rs#xavier-initialization

/// The formula-derived scale for one [fan-in, fan-out] matrix.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct XavierScale {
    fan_in: usize,
    fan_out: usize,
    target_variance: f64,
    standard_deviation: f64,
    uniform_limit: f64,
}

impl XavierScale {
    pub const fn fan_in(self) -> usize {
        self.fan_in
    }

    pub const fn fan_out(self) -> usize {
        self.fan_out
    }

    pub const fn target_variance(self) -> f64 {
        self.target_variance
    }

    pub const fn standard_deviation(self) -> f64 {
        self.standard_deviation
    }

    pub const fn uniform_limit(self) -> f64 {
        self.uniform_limit
    }
}

/// Calculates the Xavier variance, standard deviation, and uniform bound.
pub fn xavier_scale(fan_in: usize, fan_out: usize) -> Result<XavierScale, InitializationError> {
    if fan_in == 0 {
        return Err(InitializationError::ZeroFanIn);
    }
    if fan_out == 0 {
        return Err(InitializationError::ZeroFanOut);
    }
    let fan_sum = fan_in
        .checked_add(fan_out)
        .ok_or(InitializationError::FanSumOverflow { fan_in, fan_out })?;
    let target_variance = 2.0 / fan_sum as f64;
    Ok(XavierScale {
        fan_in,
        fan_out,
        target_variance,
        standard_deviation: target_variance.sqrt(),
        uniform_limit: (6.0 / fan_sum as f64).sqrt(),
    })
}

fn checked_element_count(fan_in: usize, fan_out: usize) -> Result<usize, InitializationError> {
    fan_in
        .checked_mul(fan_out)
        .ok_or(InitializationError::ShapeProductOverflow { fan_in, fan_out })
}

pub(crate) fn validate_name(name: &str) -> Result<(), InitializationError> {
    if name.is_empty() {
        return Err(InitializationError::EmptyName);
    }

    let mut previous_was_dot = true;
    for (index, byte) in name.bytes().enumerate() {
        if byte == b'.' {
            if previous_was_dot {
                return Err(InitializationError::EmptyNameSegment { index });
            }
            previous_was_dot = true;
            continue;
        }

        if !(byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'_') {
            return Err(InitializationError::InvalidNameCharacter { index, byte });
        }
        previous_was_dot = false;
    }

    if previous_was_dot {
        return Err(InitializationError::EmptyNameSegment { index: name.len() });
    }
    Ok(())
}

fn initialized_values(
    rng: &mut SplitMix64,
    scale: XavierScale,
) -> Result<Vec<f64>, InitializationError> {
    let elements = checked_element_count(scale.fan_in, scale.fan_out)?;
    let mut values = Vec::new();
    values
        .try_reserve_exact(elements)
        .map_err(|_| InitializationError::AllocationFailed { elements })?;
    for _ in 0..elements {
        let centered = 2.0 * rng.next_unit_f64() - 1.0;
        values.push(scale.uniform_limit * centered);
    }
    Ok(values)
}

Later work joined residual design and initialization choices. DeepNet (2022)

DeepNet introduces DeepNorm, which modifies the residual connection and accompanies it with theoretically derived initialization. The course’s selective initialization factor does not reproduce that method, its constants or its results.

Its method depends on that Transformer setting. Our selected $1/\sqrt L$ factor also carries no DeepNorm stability theorem.

The runnable local contrast retains one actual Xavier-initialized base set. It copies that set into an unscaled control, then completes selective initialization from the same preparation. The comparison changes the chosen initialization policy without adding a second random sample. Neither model is a DeepNorm implementation.

Copy one actual base parameter set into an unscaled control, then initialize selected decoder projection weights from the same preparation. rust/demos/practical-ch06-depth-stable-decoder/src/lib.rs#depth-same-base-control

fn fixture(id: &str, profile: &str, config: DecoderModelConfig) -> Result<Value, Box<dyn Error>> {
    let mut rng = SplitMix64::from_seed(SEED);
    let fresh = FreshDepthInitialization::prepare(
        config,
        DepthInitializationBudget {
            max_parameters: if profile == "reference-ci" {
                1188
            } else {
                8304
            },
            max_positions: if profile == "reference-ci" { 4 } else { 16 },
        },
        &mut rng,
    )?;
    // Copy the one actual base set; no second initialization or random replay.
    let base_parameters = fresh
        .base_model()
        .parameters()
        .iter()
        .map(|p| NamedParameter::from_tensor(p.name(), p.tensor().value_snapshot()))
        .collect::<Result<Vec<_>, _>>()?;
    let control = DecoderModel::from_parameters(config, base_parameters)?;
    let initialized = fresh.complete()?;
    let scaled = initialized.model();
    let alpha = initialized.record().factor();
    let mut selected = Vec::new();
    let mut unchanged = 0;
    for (base, actual) in control.parameters().iter().zip(scaled.parameters()) {
        if base.name() != actual.name() || base.tensor().shape() != actual.tensor().shape() {
            return Err("depth transform changed the parameter layout".into());
        }
        let is_selected = base.name().ends_with(".attention.output.weight")
            || base.name().ends_with(".ffn.down.weight");
        for (&before, &after) in base
            .tensor()
            .value()
            .as_slice()
            .iter()
            .zip(actual.tensor().value().as_slice())
        {
            let expected = if is_selected && config.layers() != 1 {
                before * alpha
            } else {
                before
            };
            if expected.to_bits() != after.to_bits() {
                return Err("selective stored-weight transform differed".into());
            }
        }
        if is_selected {
            selected.push(base.name().to_owned());
        } else {
            unchanged += 1;
        }
    }
    let (scaled_loss, scaled_health) = health(scaled)?;
    let (base_loss, base_health) = health(&control)?;
    if !scaled_health.within_band() {
        return Err(format!("{id}: fixed initialized fixture left the frozen health band").into());
    }
    Ok(
        json!({"id": id, "profile": profile, "vocabulary": config.vocabulary_size(),
        "width": config.model_width(), "heads": config.heads(), "ff_width": config.feed_forward_width(),
        "layers": config.layers(), "max_positions": config.max_positions(),
        "parameters": scaled.parameter_count(), "parameter_tensors": scaled.parameters().len(),
        "factor": alpha, "policy_id": initialized.record().policy_id(),
        "base_initializer_id": initialized.record().base_initializer_id(),
        "initial_rng_state": initialized.record().initial_rng_state().to_string(),
        "final_rng_state": initialized.record().final_rng_state().to_string(),
        "transformed_tensors": initialized.record().transformed_tensors(),
        "selected_names": selected, "unselected_tensors_unchanged": unchanged,
        "initialized": report_json(scaled_loss, &scaled_health),
        "same_base_unscaled_control": report_json(base_loss, &base_health)}),
    )
}

A decoder needs an explicit initialization policy as its block count grows, together with observations that distinguish residual magnitude from loss sensitivity.

Read the observations in residual-site order

The narrow fixtures all have width $4$ , one attention head, feed-forward width $4$ and position capacity $4$ . They use one, two or eight complete blocks. The CPU bridge fixture has width $16$ , $2$ heads, feed-forward width $20$ , $2$ blocks and position capacity $16$ . All use vocabulary size $266$ and the four valid positions given above. Query and key/value head counts are equal in these executed decoders.

Executed decoder parameter counts and observed residual-site counts
Decoder fixture	Parameter coordinates	Residual sites
Narrow, one block	$1188$	$3$
Narrow, two blocks	$1308$	$5$
Narrow, eight blocks	$2028$	$17$
CPU bridge, two blocks	$8304$	$5$

Read site $0$ as the input, each odd site as the state after an attention addition, and each following even site as the state after a feed-forward addition. Activation ratios use the input activation RMS; adjoint ratios use that model’s terminal residual-adjoint RMS. Each model has its own normalizers, so comparing ratios is not a comparison of absolute derivative magnitudes.

The full trace places every initialized activation and adjoint ratio inside $[0.01,100]$ for these four fixtures. Every same-base unscaled control also lies inside the band. The tables show the maximum of each ratio over that model’s sites, with displays rounded to six decimal places. Classification uses the unrounded values.

Maximum activation RMS divided by input activation RMS; fixed seed 4701
Decoder fixture	Selectively initialized	Same-base unscaled
Narrow, one block	$10.582491$	$10.582491$
Narrow, two blocks	$7.412484$	$10.582491$
Narrow, eight blocks	$12.128401$	$33.368741$
CPU bridge, two blocks	$10.869803$	$15.782601$

Maximum residual-adjoint RMS divided by terminal residual-adjoint RMS; fixed seed 4701
Decoder fixture	Selectively initialized	Same-base unscaled
Narrow, one block	$18.953075$	$18.953075$
Narrow, two blocks	$16.167999$	$20.642290$
Narrow, eight blocks	$10.196326$	$31.607222$
CPU bridge, two blocks	$18.767830$	$25.913805$

In these fixed deeper fixtures, selective initialization lowers the displayed maximum activation and adjoint ratios compared with their same-base controls. One block follows the unchanged path. These observations say only what happened for these untrained models, supplied IDs and seed. Both policies pass this finite band; the results establish neither convergence nor absolute gradient quality, nor behavior at other seeds, depths, numeric types or backends.

Keep synthetic threshold examples separate

The example also passes synthetic arrays directly to classify_ratio. Activation ratios $[1,1.2,8,101]$ have an upper failure at forward site $3$ . Gradient ratios $[0.001,0.5,0.8,1]$ have a lower failure at site $0$ . The boundary array $[0.01,1,100]$ is entirely within the inclusive band. These arrays explain the classifier; they are not measurements from an initialized decoder.

Initialized projection weights and observed residual states

The factor initializes only stored attention output and feed-forward down weights. The residual skip route enters each addition without a depth multiplier. Each fixture lists the input, then the state after every attention and feed-forward addition. Between neighboring blocks, the state after one block's feed-forward addition is reused as the next block's input; that shared residual state is listed once. Activation RMS divides by input activation RMS. Fully accumulated mean-loss adjoint RMS divides by terminal residual-adjoint RMS before final RMSNorm and the tied head. The inclusive band classifies unrounded ratios. All fixed initialized fixtures and their same-base unscaled controls are within the band. The separate synthetic arrays illustrate classifier thresholds; they are not decoder measurements.

Within each complete decoder block

The complete-block count of the whole decoder sets the factor. The base matrix comes from Xavier initialization; its transformed values are stored once before forward. The topology shows one block of the stack.

$\alpha=L^{-1/2},\quad L>0$

Zero-based block index: $b$

Block input residual state

$x_b$

The residual skip route contributes its current state directly to each addition. The initialization policy applies no multiplier to that route.
Attention residual addition

The attention branch reads the normalized block input.

Attention branch output: $a_b$

Initialize the stored attention output projection weights.

$W_{\mathrm{stored}}=\alpha W_{\mathrm{base}}$

$u_b=x_b+a_b$
Feed-forward residual addition

The feed-forward branch reads the normalized state after the attention addition.

Feed-forward branch output: $f_b$

Initialize the stored feed-forward down projection weights.

$W_{\mathrm{stored}}=\alpha W_{\mathrm{base}}$

$x_{b+1}=u_b+f_b$

Query, key, value, gate and up weights, embeddings and their tied vocabulary head, and norm gains retain their base values. Forward uses the initialized stored weights directly, without a second factor.

Observed initialized decoder residual sites

These untrained decoder measurements use the four supplied valid positions and the existing valid-target mean cross entropy, in nats per valid target. Query and key/value head counts are equal in the executed models.

Initialization seed: 4701

Inclusive band for dimensionless residual ratios: $[0.01,100]$

Activation RMS measures magnitude about zero over valid residual features. Each activation ratio divides by that model's input activation RMS and is dimensionless.

Adjoint RMS uses completed derivatives of the mean loss. Each adjoint ratio divides by that model's terminal residual-adjoint RMS before final RMSNorm and the tied head, whose downstream operations contribute to that derivative. Both normalizers must be positive and finite, without epsilon repair.

RMS, factor and ratio displays are rounded to six decimal places. Classification uses unrounded values, including both band endpoints.

Narrow decoder, one complete block

Complete decoder blocks: $1$
Residual features per position: $4$
Equal query and key/value head count: $1$
Feed-forward features: $4$
Vocabulary entries: $266$
Position capacity: $4$
Stored parameter coordinates: $1188$
Initialization factor for stored weights: $1.000000$

Input activation RMS normalizer $A_0$: $0.079065$
Terminal residual-adjoint RMS normalizer $G_{2L}$: $0.039061$

All site ratios within band

Valid-feature residual activation and completed mean-loss adjoint observations: Narrow decoder, one complete block
Residual site and stage	Absolute activation RMS $A_s$	Activation ratio and band classification $A_s/A_0$	Absolute completed mean-loss adjoint RMS $G_s$	Adjoint ratio and band classification $G_s/G_{2L}$
$0$ Input residual	$0.079065$	$1.000000$ Within band	$0.740319$	$18.953075$ Within band
$1$ Zero-based block index $0$ After attention addition	$0.655969$	$8.296554$ Within band	$0.063402$	$1.623159$ Within band
$2$ Zero-based block index $0$ After feed-forward addition	$0.836707$	$10.582491$ Within band	$0.039061$	$1.000000$ Within band

Narrow decoder, two complete blocks

Complete decoder blocks: $2$
Residual features per position: $4$
Equal query and key/value head count: $1$
Feed-forward features: $4$
Vocabulary entries: $266$
Position capacity: $4$
Stored parameter coordinates: $1308$
Initialization factor for stored weights: $0.707107$

Input activation RMS normalizer $A_0$: $0.079065$
Terminal residual-adjoint RMS normalizer $G_{2L}$: $0.039301$

All site ratios within band

Valid-feature residual activation and completed mean-loss adjoint observations: Narrow decoder, two complete blocks
Residual site and stage	Absolute activation RMS $A_s$	Activation ratio and band classification $A_s/A_0$	Absolute completed mean-loss adjoint RMS $G_s$	Adjoint ratio and band classification $G_s/G_{2L}$
$0$ Input residual	$0.079065$	$1.000000$ Within band	$0.635426$	$16.167999$ Within band
$1$ Zero-based block index $0$ After attention addition	$0.473666$	$5.990828$ Within band	$0.058492$	$1.488281$ Within band
$2$ Zero-based block index $0$ After feed-forward addition	$0.586070$	$7.412484$ Within band	$0.030488$	$0.775754$ Within band
$3$ Zero-based block index $1$ After attention addition	$0.547696$	$6.927136$ Within band	$0.036005$	$0.916137$ Within band
$4$ Zero-based block index $1$ After feed-forward addition	$0.530220$	$6.706111$ Within band	$0.039301$	$1.000000$ Within band

Narrow decoder, eight complete blocks

Complete decoder blocks: $8$
Residual features per position: $4$
Equal query and key/value head count: $1$
Feed-forward features: $4$
Vocabulary entries: $266$
Position capacity: $4$
Stored parameter coordinates: $2028$
Initialization factor for stored weights: $0.353553$

Input activation RMS normalizer $A_0$: $0.079065$
Terminal residual-adjoint RMS normalizer $G_{2L}$: $0.020722$

All site ratios within band

Valid-feature residual activation and completed mean-loss adjoint observations: Narrow decoder, eight complete blocks
Residual site and stage	Absolute activation RMS $A_s$	Activation ratio and band classification $A_s/A_0$	Absolute completed mean-loss adjoint RMS $G_s$	Adjoint ratio and band classification $G_s/G_{2L}$
$0$ Input residual	$0.079065$	$1.000000$ Within band	$0.211289$	$10.196326$ Within band
$1$ Zero-based block index $0$ After attention addition	$0.256853$	$3.248621$ Within band	$0.059247$	$2.859128$ Within band
$2$ Zero-based block index $0$ After feed-forward addition	$0.290148$	$3.669726$ Within band	$0.034452$	$1.662566$ Within band
$3$ Zero-based block index $1$ After attention addition	$0.236960$	$2.997021$ Within band	$0.026017$	$1.255532$ Within band
$4$ Zero-based block index $1$ After feed-forward addition	$0.216988$	$2.744412$ Within band	$0.024558$	$1.185095$ Within band
$5$ Zero-based block index $2$ After attention addition	$0.511366$	$6.467642$ Within band	$0.018467$	$0.891187$ Within band
$6$ Zero-based block index $2$ After feed-forward addition	$0.537408$	$6.797026$ Within band	$0.018383$	$0.887143$ Within band
$7$ Zero-based block index $3$ After attention addition	$0.758924$	$9.598714$ Within band	$0.015413$	$0.743816$ Within band
$8$ Zero-based block index $3$ After feed-forward addition	$0.784710$	$9.924844$ Within band	$0.018159$	$0.876295$ Within band
$9$ Zero-based block index $4$ After attention addition	$0.850134$	$10.752315$ Within band	$0.016304$	$0.786791$ Within band
$10$ Zero-based block index $4$ After feed-forward addition	$0.914094$	$11.561262$ Within band	$0.016780$	$0.809786$ Within band
$11$ Zero-based block index $5$ After attention addition	$0.899559$	$11.377430$ Within band	$0.020732$	$1.000469$ Within band
$12$ Zero-based block index $5$ After feed-forward addition	$0.930840$	$11.773068$ Within band	$0.021187$	$1.022415$ Within band
$13$ Zero-based block index $6$ After attention addition	$0.913800$	$11.557545$ Within band	$0.020925$	$1.009797$ Within band
$14$ Zero-based block index $6$ After feed-forward addition	$0.958935$	$12.128401$ Within band	$0.023167$	$1.118005$ Within band
$15$ Zero-based block index $7$ After attention addition	$0.932903$	$11.799161$ Within band	$0.023380$	$1.128287$ Within band
$16$ Zero-based block index $7$ After feed-forward addition	$0.892000$	$11.281826$ Within band	$0.020722$	$1.000000$ Within band

CPU bridge decoder, two complete blocks

Complete decoder blocks: $2$
Residual features per position: $16$
Equal query and key/value head count: $2$
Feed-forward features: $20$
Vocabulary entries: $266$
Position capacity: $16$
Stored parameter coordinates: $8304$
Initialization factor for stored weights: $0.707107$

Input activation RMS normalizer $A_0$: $0.082943$
Terminal residual-adjoint RMS normalizer $G_{2L}$: $0.022125$

All site ratios within band

Valid-feature residual activation and completed mean-loss adjoint observations: CPU bridge decoder, two complete blocks
Residual site and stage	Absolute activation RMS $A_s$	Activation ratio and band classification $A_s/A_0$	Absolute completed mean-loss adjoint RMS $G_s$	Adjoint ratio and band classification $G_s/G_{2L}$
$0$ Input residual	$0.082943$	$1.000000$ Within band	$0.415241$	$18.767830$ Within band
$1$ Zero-based block index $0$ After attention addition	$0.533240$	$6.428974$ Within band	$0.055351$	$2.501706$ Within band
$2$ Zero-based block index $0$ After feed-forward addition	$0.567039$	$6.836463$ Within band	$0.037754$	$1.706389$ Within band
$3$ Zero-based block index $1$ After attention addition	$0.853538$	$10.290620$ Within band	$0.026215$	$1.184846$ Within band
$4$ Zero-based block index $1$ After feed-forward addition	$0.901577$	$10.869803$ Within band	$0.022125$	$1.000000$ Within band

The JSON also reports branch-output RMS and norm-gain vectors separately. Those diagnostics are outside the residual-site ratio classification.

Separate same-base unscaled controls

For each fixture, the unscaled control copies the one actual base parameter set before selected weights are transformed. No model is resampled. Each control uses its own normalizers. Every fixed control also lies within the residual ratio band; comparing ratios does not compare absolute adjoint magnitudes.

Same-base unscaled normalizers and dimensionless ratio ranges over all residual sites
Decoder fixture	Input activation RMS normalizer $A_0$	Terminal residual-adjoint RMS normalizer $G_{2L}$	Activation ratio range $A_s/A_0$	Adjoint ratio range $G_s/G_{2L}$	Ratio-band classification
Narrow decoder, one complete block	$0.079065$	$0.039061$	Minimum over sites: $1.000000$ Maximum over sites: $10.582491$	Minimum over sites: $1.000000$ Maximum over sites: $18.953075$	All site ratios within band
Narrow decoder, two complete blocks	$0.079065$	$0.028058$	Minimum over sites: $1.000000$ Maximum over sites: $10.582491$	Minimum over sites: $0.745982$ Maximum over sites: $20.642290$	All site ratios within band
Narrow decoder, eight complete blocks	$0.079065$	$0.007203$	Minimum over sites: $1.000000$ Maximum over sites: $33.368741$	Minimum over sites: $0.773411$ Maximum over sites: $31.607222$	All site ratios within band
CPU bridge decoder, two complete blocks	$0.082943$	$0.015205$	Minimum over sites: $1.000000$ Maximum over sites: $15.782601$	Minimum over sites: $1.000000$ Maximum over sites: $25.913805$	All site ratios within band

Separate synthetic threshold examples

These arrays are supplied directly to the ratio classifier. They illustrate threshold decisions and are separate from initialized decoder observations. The band includes both endpoints; classification uses unrounded values.

Synthetic activation ratios
Synthetic forward-site index	Supplied dimensionless ratio	Ratio-band classification
$0$	$1$	Within band
$1$	$1.2$	Within band
$2$	$8$	Within band
$3$	$101$	Above band

Synthetic gradient ratios
Synthetic forward-site index	Supplied dimensionless ratio	Ratio-band classification
$0$	$0.001$	Below band
$1$	$0.5$	Within band
$2$	$0.8$	Within band
$3$	$1$	Within band

Synthetic inclusive boundaries
Synthetic forward-site index	Supplied dimensionless ratio	Ratio-band classification
$0$	$0.01$	Within band
$1$	$1$	Within band
$2$	$100$	Within band

The laptop_plan is also separate from the executed fixtures. It contains only symbolic arithmetic and metadata: vocabulary $16384$ , width $512$ , $8$ blocks, $8$ query heads, $2$ key/value heads, feed-forward width $1536$ and context $512$ , giving $32514560$ parameters and factor $1/\sqrt8$ . It allocates no model arrays and supplies no seeded execution, measured health or performance result. Unequal-head execution remains later work.

Optional practice: inspect initialization and its limits

Use the fixed-example command above to inspect worked_values at one, two and eight blocks. Explain the selected coefficients, unchanged query coefficients and ideal Xavier half-bounds and variances.
Inspect the eight-block fixture’s ordered sites. Explain why its factor uses eight blocks while its report has seventeen sites.
Explain the stored-coordinate and base-coordinate derivatives for the scalar product with input two, and why forward needs no second factor.
Inspect synthetic_classifiers. Explain the upper, lower and inclusive outcomes, and distinguish them from fixtures.
Inspect the initialized and same-base unscaled reports. Explain what the maximum-ratio comparisons establish, why both normalizers must be usable, and why the reported ratios do not establish convergence.
Explain what happens when a fresh preparation is dropped, when its budget refuses construction, and when already initialized or restored parameters are supplied for fresh initialization. Distinguish restoration from the symbolic laptop entry.

Explanations for factors, site counts, derivatives, thresholds and construction states

The factors are $1$ , $1/\sqrt2$ and $1/\sqrt8$ . Multiplying only the selected vector gives the coefficient values explained above; the query vector stays $[0.2,-0.4,0,0.1]$ . For a selected $4\times4$ matrix, the ideal half-bounds are $\sqrt{3/4}$ , $\sqrt{3/8}$ and $\sqrt{3/32}$ , with variances $1/4$ , $1/8$ and $1/32$ . The variance changes by the square of the factor. These are distribution properties, not a finite-sample measurement.
Eight complete blocks contain $16$ residual additions. The input plus each addition’s output gives $17$ sites; a block’s final state is the next block’s input and is counted once. The divisor remains $L=8$ .
$\partial y/\partial w_{\mathrm{stored}}=2$ , while $\partial y/\partial w_{\mathrm{base}}=2\alpha$ when the transformation is retained in the graph. Our parameter is the stored coefficient. Multiplying again in forward would change the intended initialized model.
$101>100$ puts the first synthetic array’s site $3$ above the band. $0.001<0.01$ puts the second array’s site $0$ below it. Equality at $0.01$ or $100$ is allowed. The actual four initialized decoder reports lie within the band and are separate observations.
The deeper fixed initialized fixtures have lower observed maximum ratios than their same-base controls; the one-block observations coincide. All controls also pass. Activation RMS divides by input RMS and adjoint RMS by terminal residual-adjoint RMS, so each denominator must be positive and finite. Ratios omit absolute scale and cover these fixtures before training, so they establish neither adequate absolute derivatives nor convergence.
Dropping preparation leaves the caller RNG unchanged, as does refusal. Checked parameter and position bounds precede model-array allocation. Fresh initialization refuses Initialized and Restored states before draws or mutation. Restoration takes a construction record and actual stored parameters and does not rescale. The laptop entry has metadata and arithmetic only, with no parameter arrays or execution to restore.

Carry explicit construction choices into the decoder core

Depth-aware initialization changes the starting output projections. It retains the selected source pairs, local positions, occurrence-local causal ranges and valid-target mean loss from Chapter 5. More blocks must preserve those boundaries; initialization cannot repair an incorrectly joined context or target.

Chapter 7 will place decoder dimensions and construction choices in a configurable core. Carry forward the distinction between fresh initialization and exact restoration, along with the meaning and limits of these residual observations. Unequal query and key/value head counts and trained-checkpoint serialization remain later work.

The problem: depth adds more residual contributions

Start with four selected coefficients

Apply the factor once, using complete blocks

What the variance intuition assumes

Differentiate the coefficient actually stored

What the initialization symbols mean

Construct fresh parameters and observe the existing decoder

Measure residual magnitude and completed loss sensitivity

Run the fixed CPU example

History: fan-based scale and depth-aware residual methods

Read the observations in residual-site order

Keep synthetic threshold examples separate

Initialized projection weights and observed residual states

Within each complete decoder block

Block input residual state

Attention residual addition

Feed-forward residual addition

Observed initialized decoder residual sites

Narrow decoder, one complete block

Narrow decoder, two complete blocks

Narrow decoder, eight complete blocks

CPU bridge decoder, two complete blocks

Separate same-base unscaled controls

Separate synthetic threshold examples

Optional practice: inspect initialization and its limits

Carry explicit construction choices into the decoder core