Start trainable weights at a reproducible scale

A correct gradient still needs a useful starting point

Chapter 16 can differentiate a matrix transformation, but the backward pass cannot choose its initial weights. In a bias-free layer, two hidden units receiving the same input through the same activation compute the same feature when their incoming weight columns match. If the downstream calculation treats them equally, both receive the same gradient column. Applying equal updates then keeps the units identical: training has not given them separate features.

Weight scale is a second problem. Each matrix output adds contributions from several inputs, so changing the number of contributions can change the spread of the resulting signal. Repeating matrix transformations can compound that scale change. We need starting weights that distinguish the selected hidden units, account for matrix width, and can be reproduced while comparing changes to the model.

Work through one sampled projection

Use a projection of shape [2,2]: its two rows correspond to input coordinates and its two columns to output coordinates. Start the generator at seed 17. With fan-in $2$ and fan-out $2$, the Xavier-style rule targets variance $1/2$. Its standard deviation is $1/\sqrt{2}$, and its zero-centered uniform bound is $\sqrt{3/2}$. The four weights appear in row-major order, rounded to twelve decimal places:

 0.004950883736  -0.265932089217
-0.420504358848  -0.676313443233

The two columns are unequal in this sampled matrix, so they do not begin with the identical-column condition described above. That fact is a starting-point comparison, not evidence that the model has already learned useful features.

Restarting the same construction from seed 17 reproduces the stored tensor bit for bit. The same selected request from seed 18 produces a different tensor. Exact replay needs the same generator state, shape, fan values, and construction order, not just a remembered seed after other parameters have already consumed draws. The seed-18 result is one observed comparison, not a promise that every pair of seeds yields different tensors.

    let scale = xavier_scale(2, 2)?;
    let mut rng = SplitMix64::from_seed(FIXTURE_SEED);
    let projection = projection_parameter(&mut rng)?;
    let weights = projection.tensor().value_snapshot();

    let mut matching_rng = SplitMix64::from_seed(FIXTURE_SEED);
    let matching = projection_parameter(&mut matching_rng)?;
    let mut alternate_rng = SplitMix64::from_seed(ALTERNATE_SEED);
    let alternate = projection_parameter(&mut alternate_rng)?;

Let both matrix widths determine the starting spread

For the stored matrix $W$, weight $W_{ij}$ connects input coordinate $i$ to output coordinate $j$. Fan-in counts the weighted inputs summed into each output; fan-out counts the outputs receiving each input.

We estimate scale in a simplified setting: independent zero-centered dense weights with a common variance, independent zero-centered input features with a common variance, weight/input independence, and a symmetric activation near a unit-slope linear regime. The analogous backward estimate uses zero-centered arriving gradients with a common variance, independent of the weights.

Under these assumptions, the forward signal’s variance is multiplied by fan-in and weight variance. The backward gradient’s variance instead involves fan-out and weight variance. Keeping either multiplier at one asks for the reciprocal of its width; unequal widths ask for different scales.

Glorot/Xavier initialization uses this compromise:

The left side describes the target variance of the weight distribution, in squared weight units. It does not require the four displayed weights to have exactly that measured variance. The numerator balances the forward and backward requests; when both widths match, the same scale meets both idealized conditions.

To obtain weights of both signs at that spread, sample a zero-centered uniform distribution over $[-a,a)$. Its variance is $a^2/3$, so matching the target gives $a=\sqrt{6/(\operatorname{fan}_{in}+\operatorname{fan}_{out})}$. The standard deviation is the square root of the target variance. Standard deviation and bound are both in weight units, but the bound marks the interval endpoint rather than the typical spread around the mean.

SiLU does not exactly meet the activation assumption. Normalization, residual paths, data, and optimization also matter in the eventual decoder; the formula is not a decoder-wide variance-preservation theorem.

Turn the distribution into repeatable Rust samples

A bound alone does not choose individual weights. SplitMix64 supplies the repeatable draws. Its seed is the raw u64 state before the first increment, so seed zero is valid. Wrapping integer operations advance and mix that state. Dividing the integer in the high 53 mixed bits by the number of possible 53-bit values gives a binary64 fraction in $[0,1)$. A fixed state gives the same continuation. This is deterministic pseudorandom sampling, not cryptographic randomness or ideal continuous sampling.

/// A small deterministic generator with an explicit resumable 64-bit state.
///
/// This generator is suitable for reproducible teaching fixtures. It is not a
/// cryptographically secure random-number generator.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SplitMix64 {
    state: u64,
}

impl SplitMix64 {
    /// Treats `seed` as the raw state before the first increment and draw.
    pub const fn from_seed(seed: u64) -> Self {
        Self { state: seed }
    }

    /// Resumes directly from a previously recorded raw state.
    pub const fn from_state(state: u64) -> Self {
        Self { state }
    }

    /// Returns the raw state that the next draw will advance.
    pub const fn state(&self) -> u64 {
        self.state
    }

    /// Advances and mixes one exactly specified 64-bit value.
    pub fn next_u64(&mut self) -> u64 {
        self.state = self.state.wrapping_add(SPLITMIX_INCREMENT);
        let mut value = self.state;
        value = (value ^ (value >> 30)).wrapping_mul(SPLITMIX_MIX_ONE);
        value = (value ^ (value >> 27)).wrapping_mul(SPLITMIX_MIX_TWO);
        value ^ (value >> 31)
    }

    /// Maps the high 53 bits of one draw to the binary64 interval [0,1).
    pub fn next_unit_f64(&mut self) -> f64 {
        ((self.next_u64() >> 11) as f64) * BINARY64_UNIT_SCALE
    }
}

xavier_scale first rejects zero widths and an overflowing integer fan sum. It then stores the formula’s variance in target_variance, its square root in standard_deviation, and the sampling bound in uniform_limit. The sampler doubles a unit draw, subtracts one to center it, and multiplies by uniform_limit. One draw supplies each weight in row-major order, so the matrix shape fixes how many draws construction consumes.

/// The formula-derived scale for one [fan-in, fan-out] matrix.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct XavierScale {
    fan_in: usize,
    fan_out: usize,
    target_variance: f64,
    standard_deviation: f64,
    uniform_limit: f64,
}

impl XavierScale {
    pub const fn fan_in(self) -> usize {
        self.fan_in
    }

    pub const fn fan_out(self) -> usize {
        self.fan_out
    }

    pub const fn target_variance(self) -> f64 {
        self.target_variance
    }

    pub const fn standard_deviation(self) -> f64 {
        self.standard_deviation
    }

    pub const fn uniform_limit(self) -> f64 {
        self.uniform_limit
    }
}

/// Calculates the Xavier variance, standard deviation, and uniform bound.
pub fn xavier_scale(fan_in: usize, fan_out: usize) -> Result<XavierScale, InitializationError> {
    if fan_in == 0 {
        return Err(InitializationError::ZeroFanIn);
    }
    if fan_out == 0 {
        return Err(InitializationError::ZeroFanOut);
    }
    let fan_sum = fan_in
        .checked_add(fan_out)
        .ok_or(InitializationError::FanSumOverflow { fan_in, fan_out })?;
    let target_variance = 2.0 / fan_sum as f64;
    Ok(XavierScale {
        fan_in,
        fan_out,
        target_variance,
        standard_deviation: target_variance.sqrt(),
        uniform_limit: (6.0 / fan_sum as f64).sqrt(),
    })
}

fn checked_element_count(fan_in: usize, fan_out: usize) -> Result<usize, InitializationError> {
    fan_in
        .checked_mul(fan_out)
        .ok_or(InitializationError::ShapeProductOverflow { fan_in, fan_out })
}

pub(crate) fn validate_name(name: &str) -> Result<(), InitializationError> {
    if name.is_empty() {
        return Err(InitializationError::EmptyName);
    }

    let mut previous_was_dot = true;
    for (index, byte) in name.bytes().enumerate() {
        if byte == b'.' {
            if previous_was_dot {
                return Err(InitializationError::EmptyNameSegment { index });
            }
            previous_was_dot = true;
            continue;
        }

        if !(byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'_') {
            return Err(InitializationError::InvalidNameCharacter { index, byte });
        }
        previous_was_dot = false;
    }

    if previous_was_dot {
        return Err(InitializationError::EmptyNameSegment { index: name.len() });
    }
    Ok(())
}

fn initialized_values(
    rng: &mut SplitMix64,
    scale: XavierScale,
) -> Result<Vec<f64>, InitializationError> {
    let elements = checked_element_count(scale.fan_in, scale.fan_out)?;
    let mut values = Vec::new();
    values
        .try_reserve_exact(elements)
        .map_err(|_| InitializationError::AllocationFailed { elements })?;
    for _ in 0..elements {
        let centered = 2.0 * rng.next_unit_f64() - 1.0;
        values.push(scale.uniform_limit * centered);
    }
    Ok(values)
}

The scale calculations and weights use f64. Integer checks happen before conversion, but a sufficiently large valid usize count can round when cast to f64. Division, square roots, and multiplication also use represented arithmetic. The generator’s finite grid and the stored values therefore differ from an ideal continuous distribution; printing twelve decimals rounds the stored value again. Bit-for-bit replay is an implementation result, not a claim that a finite matrix’s variance must equal the mathematical target. These small examples do not exercise floating-point underflow or overflow; the checked overflow errors concern integer dimensions.

Read the widths as counts of connections

The formula uses input rows and output columns, matching the projection’s stored shape. These local meanings distinguish the two widths:

For fan-in $2$ and fan-out $2$, standard deviation is $0.707106781187$ and uniform bound is $1.224744871392$. Increase fan-in to $4$ while keeping fan-out $2$: the target variance becomes $1/3$, standard deviation $0.577350269190$, and bound $1.000000000000$. More input contributions require a smaller starting spread under the compromise. Width must therefore be known before sampling.

Keep parameter construction replayable

A sampled matrix must become a trainable TensorValue leaf so later backward passes can accumulate its gradients. NamedParameter also gives that leaf a stable external name, such as decoder.block.0.attention.query.weight. The name identifies the parameter’s role across reconstruction; it is not the runtime node itself. Cloning a parameter shares its tape leaf, while constructing a new leaf with equal values does not.

Names need a fixed grammar and collections need duplicate rejection so name lookup cannot silently select an ambiguous parameter. Declaration order stays deliberate: on one shared generator stream, each successful construction consumes the draws that precede the next parameter. Inserting an earlier matrix changes the later samples.

A rejected request must not shift that stream. Name and width checks reject invalid requests before drawing, and checked dimension arithmetic prevents a wrapped storage count. Sampling uses a cloned trial generator; the caller’s state is committed only after tensor and trainable-leaf construction succeed. Every returned sampler error therefore leaves the caller’s state unchanged. Correcting a failed request can reuse its intended starting state.

/// One immutable external name paired with one trainable tensor-tape leaf.
#[derive(Clone, Debug)]
pub struct NamedParameter {
    name: String,
    tensor: TensorValue,
}

impl NamedParameter {
    /// Wraps an already-created tensor as a named trainable leaf.
    pub fn from_tensor(
        name: impl Into<String>,
        tensor: Tensor,
    ) -> Result<Self, InitializationError> {
        let name = name.into();
        validate_name(&name)?;
        Ok(Self {
            name,
            tensor: TensorValue::parameter(tensor)?,
        })
    }

    /// Samples one [fan-in, fan-out] trainable matrix transactionally.
    pub fn xavier_uniform(
        name: impl Into<String>,
        fan_in: usize,
        fan_out: usize,
        rng: &mut SplitMix64,
    ) -> Result<Self, InitializationError> {
        let name = name.into();
        validate_name(&name)?;
        let scale = xavier_scale(fan_in, fan_out)?;
        let elements = checked_element_count(fan_in, fan_out)?;

        let mut trial = rng.clone();
        let values = initialized_values(&mut trial, scale)?;
        debug_assert_eq!(values.len(), elements);
        let tensor = Tensor::from_vec(vec![fan_in, fan_out], values)?;
        let parameter = Self {
            name,
            tensor: TensorValue::parameter(tensor)?,
        };
        *rng = trial;
        Ok(parameter)
    }

    /// Returns the stable external identity used by layers and checkpoints.
    pub fn name(&self) -> &str {
        &self.name
    }

    /// Borrows the trainable tape leaf without duplicating its tensor storage.
    pub fn tensor(&self) -> &TensorValue {
        &self.tensor
    }
}

/// A duplicate-checked, declaration-ordered set of named parameters.
#[derive(Clone, Debug, Default)]
pub struct NamedParameters {
    parameters: Vec<NamedParameter>,
}

impl NamedParameters {
    pub fn try_new(parameters: Vec<NamedParameter>) -> Result<Self, InitializationError> {
        for repeated in 0..parameters.len() {
            if let Some(first) = parameters[..repeated]
                .iter()
                .position(|parameter| parameter.name() == parameters[repeated].name())
            {
                return Err(InitializationError::DuplicateName {
                    name: parameters[repeated].name().to_owned(),
                    first,
                    repeated,
                });
            }
        }
        Ok(Self { parameters })
    }

    pub fn len(&self) -> usize {
        self.parameters.len()
    }

    pub fn is_empty(&self) -> bool {
        self.parameters.is_empty()
    }

    pub fn as_slice(&self) -> &[NamedParameter] {
        &self.parameters
    }

    pub fn iter(&self) -> impl ExactSizeIterator<Item = &NamedParameter> {
        self.parameters.iter()
    }

    pub fn get(&self, name: &str) -> Option<&NamedParameter> {
        self.parameters
            .iter()
            .find(|parameter| parameter.name() == name)
    }
}

The parameter kinds need not share one starting value. Our later decoder uses Xavier-style uniform matrix weights, optional zero biases, and RMSNorm gains of one. Zero biases add no initial offset, and unit gains apply no additional feature scaling after normalization. The token table reuses the sampler with vocabulary size and feature width as its shape inputs; that convention does not follow from a variance calculation for selecting a row.

From random word features to width-aware repeated projections

The earlier language-model example is Bengio et al., A Neural Probabilistic Language Model. Bengio et al. jointly train a word-feature matrix and neural matrices for next-word prediction. They report initializing the word features randomly, as for neural-network weights.

Random word features gave early neural language models trainable starting values, but randomness alone does not choose a scale for each matrix width. Bengio et al. do not specify a dimension-aware reproducible initialization rule; repeated transformations make that missing scale choice more consequential.

Glorot and Bengio connect starting weight variance to both forward and backward matrix widths under near-linear and independence assumptions. Transformers later combine learned embeddings with repeated attention and feed-forward projections, so many separately shaped trainable matrices need starting values.

The variance-aware analysis is Glorot and Bengio, Understanding the difficulty of training deep feedforward neural networks. Glorot and Bengio derive a compromise between fan-in and fan-out variance conditions under simplifying assumptions: target weight variance is 2 divided by the sum of the widths, implemented with a normalized zero-centered uniform distribution.

The later architecture is Vaswani et al., Attention Is All You Need. Vaswani et al. use learned embeddings and repeat query, key, value, attention-output, and two feed-forward projections in Transformer layers. Their paper does not prescribe a parameter initializer.

Our decoder uses reproducible Xavier-style uniform matrix samples, stable parameter names, zero optional biases, and RMSNorm gains of one. Its token table reuses the sampler by matrix shape. These are explicit construction policies: the Transformer paper does not prescribe them, and they do not guarantee exact variance preservation through SiLU, normalization, or residual paths.

The Transformer’s attention-score and embedding scaling act during forward computation; they are not instructions for choosing initial weights. Likewise, the papers do not define our raw-seed mapping, name grammar, construction order, or error policy. The Rust contrast below isolates an initialization failure condition without claiming to reproduce the historical models.

Inspect equal gradients and construction guarantees

The sampled projection avoids identical columns in the selected request. The zero-weight contrast shows why that distinction matters. Set $x=[1,-1]$, use a zero [2,2] input matrix, apply SiLU, and combine the two hidden values with equal output weights [1,1]. Both hidden values and the scalar output are zero.

The reverse pass is seeded with one at that scalar output. Equal output weights give both hidden units the same upstream factor. SiLU’s derivative at zero is $1/2$, and each input weight multiplies that factor by its input coordinate. Both gradient columns are therefore $[0.5,-0.5]^\top$, stored row-major as [0.5,0.5,-0.5,-0.5]. The backward pass produces a nonzero gradient, yet equal updates keep the two columns equal.

pub fn zero_symmetry_probe() -> Result<SymmetryProbe, Box<dyn Error>> {
    let input = TensorValue::constant(tensor(&[1, 2], &[1.0, -1.0])?)?;
    let input_weights = TensorValue::parameter(tensor(&[2, 2], &[0.0; 4])?)?;
    let output_weights = TensorValue::constant(tensor(&[2, 1], &[1.0, 1.0])?)?;

    let hidden = input.matmul(&input_weights)?.silu()?;
    let output = hidden.matmul(&output_weights)?;
    let seed = tensor(&[1, 1], &[1.0])?;
    output.backward_with_seed(&seed.view(), GraphRetention::Retain)?;

    let output_value = output.value().as_slice()[0];
    let gradient = input_weights
        .gradient_snapshot()
        .expect("the input weights are a trainable leaf");
    let columns_equal = gradient.as_slice()[0] == gradient.as_slice()[1]
        && gradient.as_slice()[2] == gradient.as_slice()[3];
    Ok(SymmetryProbe {
        output: output_value,
        gradient,
        columns_equal,
    })
}

The condition is equal treatment, not simply the presence of a zero. Different downstream treatment would remove this particular equality argument. Zero optional biases and unit normalization gains are separate choices; they do not make unequal sampled feature columns identical.

Reject an invalid request without changing later draws

Valid names contain nonempty dot-separated segments of lowercase ASCII letters, digits, and underscores. The constructor checks the name first, then zero fan-in, zero fan-out, fan-sum overflow, shape-product overflow, storage reservation, tensor construction, and trainable-leaf construction. This order makes the first returned error stable when several inputs are invalid. The guarantee concerns returned errors; it does not promise recovery from every possible process allocation abort.

/// A deterministic rejection while constructing or collecting a parameter.
#[derive(Clone, Debug, PartialEq)]
pub enum InitializationError {
    EmptyName,
    EmptyNameSegment {
        index: usize,
    },
    InvalidNameCharacter {
        index: usize,
        byte: u8,
    },
    ZeroFanIn,
    ZeroFanOut,
    FanSumOverflow {
        fan_in: usize,
        fan_out: usize,
    },
    ShapeProductOverflow {
        fan_in: usize,
        fan_out: usize,
    },
    AllocationFailed {
        elements: usize,
    },
    DuplicateName {
        name: String,
        first: usize,
        repeated: usize,
    },
    Tensor(TensorError),
    Autodiff(TensorAutodiffError),
}

impl fmt::Display for InitializationError {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::EmptyName => formatter.write_str("parameter name must not be empty"),
            Self::EmptyNameSegment { index } => write!(
                formatter,
                "parameter name has an empty dot-separated segment at byte {index}"
            ),
            Self::InvalidNameCharacter { index, byte } => write!(
                formatter,
                "parameter name byte {index} must be lowercase ASCII, a digit, underscore, or dot; got 0x{byte:02x}"
            ),
            Self::ZeroFanIn => formatter.write_str("fan-in must be greater than zero"),
            Self::ZeroFanOut => formatter.write_str("fan-out must be greater than zero"),
            Self::FanSumOverflow { fan_in, fan_out } => write!(
                formatter,
                "fan-in {fan_in} plus fan-out {fan_out} does not fit usize"
            ),
            Self::ShapeProductOverflow { fan_in, fan_out } => write!(
                formatter,
                "matrix shape [{fan_in},{fan_out}] does not fit usize"
            ),
            Self::AllocationFailed { elements } => write!(
                formatter,
                "could not reserve storage for {elements} initialized values"
            ),
            Self::DuplicateName {
                name,
                first,
                repeated,
            } => write!(
                formatter,
                "parameter name {name:?} first appears at index {first} and repeats at index {repeated}"
            ),
            Self::Tensor(error) => error.fmt(formatter),
            Self::Autodiff(error) => error.fmt(formatter),
        }
    }
}

impl Error for InitializationError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        match self {
            Self::Tensor(error) => Some(error),
            Self::Autodiff(error) => Some(error),
            _ => None,
        }
    }
}

impl From<TensorError> for InitializationError {
    fn from(error: TensorError) -> Self {
        Self::Tensor(error)
    }
}

impl From<TensorAutodiffError> for InitializationError {
    fn from(error: TensorAutodiffError) -> Self {
        Self::Autodiff(error)
    }
}

NamedParameter::from_tensor offers the same named trainable-leaf boundary for manually supplied finite values, checking the name before the leaf. NamedParameters::try_new scans declaration order and reports the earliest repeated name together with its first index; it does not sort or silently merge parameters.

The example constructs the query projection first and then the [4,2] token table on the continuing generator stream. A clone of the projection shares the original leaf. The independently replayed equal projection has the same values but a different runtime leaf.

    let token_table = NamedParameter::xavier_uniform("token_embedding.weight", 4, 2, &mut rng)?;
    let parameters = NamedParameters::try_new(vec![projection.clone(), token_table])?;

    let projection_clone = projection.clone();
    let clone_same_node = projection.tensor().is_same_node(projection_clone.tensor());
    let recreated_same_node = projection.tensor().is_same_node(matching.tensor());

Decoder.weight is rejected because its uppercase byte violates the name grammar. The valid name decoder.invalid.weight with zero fan-in instead returns ZeroFanIn. Collecting two projection clones repeats their name; duplicate rejection is a collection operation and consumes no draws. The example confirms that the rejected sampler calls leave the continuing generator state unchanged.

    let state_before_errors = rng.state();
    let invalid_name =
        NamedParameter::xavier_uniform("Decoder.weight", 2, 2, &mut rng).unwrap_err();
    let zero_fan_in =
        NamedParameter::xavier_uniform("decoder.invalid.weight", 0, 2, &mut rng).unwrap_err();
    let duplicate_name =
        NamedParameters::try_new(vec![projection.clone(), projection.clone()]).unwrap_err();
    let rng_unchanged = rng.state() == state_before_errors;

Run the complete example from the repository root:

./course run cargo run --quiet --locked -p ch17-parameter-initialization

Its output reports the same scale and four sampled weights explained above, then the zero-symmetry result, names, leaf identities, and rejected-request evidence. The decimal weight display is rounded; the replay comparison uses stored tensor values.

    let report = learner_report()?;

    println!("seed: 17");
    println!("projection: shape=2x2 fan_in=2 fan_out=2");
    println!("target variance: {}", fixed(report.scale.target_variance()));
    println!("uniform limit: {}", fixed(report.scale.uniform_limit()));
    println!(
        "weights: {}",
        report
            .weights
            .as_slice()
            .iter()
            .map(|value| fixed(*value))
            .collect::<Vec<_>>()
            .join(",")
    );
    println!("same seed reproduces: {}", report.same_seed_reproduces);
    println!("different seed differs: {}", report.different_seed_differs);
    println!(
        "zero symmetry: output={} columns-equal={} gradient={}",
        fixed(report.symmetry.output),
        report.symmetry.columns_equal,
        report
            .symmetry
            .gradient
            .as_slice()
            .iter()
            .map(|value| fixed(*value))
            .collect::<Vec<_>>()
            .join(",")
    );
    println!(
        "parameters: {}",
        report
            .parameters
            .iter()
            .map(|parameter| format!(
                "{}[{}]",
                parameter.name(),
                shape(&parameter.tensor().shape())
            ))
            .collect::<Vec<_>>()
            .join(" | ")
    );
    println!(
        "identity: clone-same-node={} recreated-same-node={}",
        report.clone_same_node, report.recreated_same_node
    );
    println!(
        "validation: invalid-name | duplicate-name | zero-fan-in; rng-unchanged={}",
        report.rng_unchanged
    );
    println!("chapter 18 handoff: initialize a trainable token table");

Separate measured weights from expected linear variance

Four weights make replay and unequal columns easy to inspect, but they do not show a distribution’s spread clearly. The figure uses three [64,64] matrices with $4096$ weights each. The zero matrix uses no draws. The oversized matrix doubles every seed-17 Xavier weight, so the two uniform matrices have the same base draws and differ only in scale. Their different histograms therefore isolate the effect of the doubled bound.

All three populations use common equal-width bins. Each cell reports a count and its percentage of the complete matrix. The measured mean and population variance come from those finite weights: the calculation finds the mean first, then divides the sum of squared deviations by the weight count. A sampled mean need not be exactly zero, and the measured variance need not equal the target.

The depth table uses a different kind of evidence. It starts with independent zero-centered input features of common unit variance before any layer and follows expected signal variance through four independent linear layers using the target weight variances. Each layer has equal input and output widths and assumes independent zero-centered weights with a common variance, independent of its inputs. Doubling the Xavier bound gives four times the weight variance, so each such layer multiplies expected signal variance by four. Zero collapses the signal variance; equal-width Xavier keeps the expected variance at one. These entries are not measurements from a forward pass through the sampled matrices or guarantees for the nonlinear residual decoder.

pub fn render_trace() -> Result<String, Box<dyn Error>> {
    let scale = xavier_scale(WIDTH, WIDTH)?;
    let mut rng = SplitMix64::from_seed(FIXTURE_SEED);
    let xavier = NamedParameter::xavier_uniform("diagnostic.weight", WIDTH, WIDTH, &mut rng)?
        .tensor()
        .value()
        .as_slice()
        .to_vec();
    let oversized: Vec<_> = xavier.iter().map(|value| value * 2.0).collect();
    let zero = vec![0.0; SAMPLE_COUNT];

    let zero_statistics = statistics(&zero);
    let oversized_statistics = statistics(&oversized);
    let xavier_statistics = statistics(&xavier);
    let zero_histogram = histogram(&zero);
    let oversized_histogram = histogram(&oversized);
    let xavier_histogram = histogram(&xavier);

    let mut same_seed_rng = SplitMix64::from_seed(FIXTURE_SEED);
    let same_seed =
        NamedParameter::xavier_uniform("diagnostic.weight", WIDTH, WIDTH, &mut same_seed_rng)?
            .tensor()
            .value()
            .as_slice()
            .to_vec();
    let mut alternate_rng = SplitMix64::from_seed(ALTERNATE_SEED);
    let alternate =
        NamedParameter::xavier_uniform("diagnostic.weight", WIDTH, WIDTH, &mut alternate_rng)?
            .tensor()
            .value()
            .as_slice()
            .to_vec();

    let mut trace = String::new();
    writeln!(trace, "TRACE parameter-initialization-v2 BEGIN")?;
    writeln!(
        trace,
        "FIXTURE name=fixed-seed-width64 generator=splitmix64 mapping=top53-affine seed=17 shape=64x64 samples=4096 fan-in=64 fan-out=64 statistic=population-two-pass layers=0,1,2,3,4 propagation=expected-linear-independent input-variance=1.000000000000 display-input-variance=1"
    )?;
    writeln!(
        trace,
        "BINNING edges={} display-edges={} width=0.100000000000 display-width=0.10 closure=left-closed-right-open-last-closed",
        fixed_list(&EDGES),
        DISPLAY_EDGES.join(",")
    )?;
    writeln!(
        trace,
        "{}",
        distribution_line("zero", "none", 0.0, &zero_statistics)
    )?;
    writeln!(trace, "{}", histogram_line("zero", &zero_histogram))?;
    writeln!(
        trace,
        "{}",
        distribution_line(
            "oversized",
            "17",
            scale.uniform_limit() * 2.0,
            &oversized_statistics,
        )
    )?;
    writeln!(
        trace,
        "{}",
        histogram_line("oversized", &oversized_histogram)
    )?;
    writeln!(
        trace,
        "{}",
        distribution_line("xavier", "17", scale.uniform_limit(), &xavier_statistics)
    )?;
    writeln!(trace, "{}", histogram_line("xavier", &xavier_histogram))?;
    writeln!(
        trace,
        "PAIRING seed=17 base-draws-equal=yes oversized-to-xavier-limit=2.000000000000"
    )?;
    writeln!(
        trace,
        "PROPAGATION kind=zero variances={} display-variances={}",
        fixed_list(&expected_variances(0.0)),
        display_integer_list(&expected_variances(0.0))
    )?;
    writeln!(
        trace,
        "PROPAGATION kind=oversized variances={} display-variances={}",
        fixed_list(&expected_variances(4.0)),
        display_integer_list(&expected_variances(4.0))
    )?;
    writeln!(
        trace,
        "PROPAGATION kind=xavier variances={} display-variances={}",
        fixed_list(&expected_variances(1.0)),
        display_integer_list(&expected_variances(1.0))
    )?;
    writeln!(
        trace,
        "REPRODUCIBILITY seed=17 same-seed-equal={} alternate-seed=18 alternate-seed-different={}",
        if xavier == same_seed { "yes" } else { "no" },
        if xavier != alternate { "yes" } else { "no" },
    )?;
    writeln!(trace, "TRACE parameter-initialization-v2 END")?;
    Ok(trace)
}

Read the histogram as measured spread in one finite population and the depth table as an assumption-bound scale argument. Pairing the draws makes the scale comparison controlled; keeping the two kinds of variance separate prevents the picture from implying a decoder-wide theorem.

Optional practice: reproduce and inspect the explained results

These tasks revisit results already explained in the lesson. They are optional; the worked solution and its limits do not depend on completing them.

1. Reproduce the target variance, standard deviation, and uniform bound for fan-in $2$ and fan-out $2$.
2. Recalculate the target variance, standard deviation, and uniform bound for fan-in $4$ and fan-out $2$, and explain the reduced starting scale.
3. Inspect the zero-weight SiLU graph and derive the two input-weight gradient columns from the equal output weights and input signs.
4. Reproduce the same seed, shape, fan values, and construction order; compare the stored tensor values rather than only printed decimals.
5. Inspect the selected seed-18 tensor comparison with seed 17 for shape [2,2], fan-in $2$, and fan-out $2$; explain why the observed stored-weight difference does not establish uniqueness for every possible seed.
6. Explain why one finite [2,2] tensor need not have measured variance exactly $1/2$.
7. Inspect the query-projection/token-table collection; identify its declaration order and distinguish a clone from an equal reconstruction.
8. Inspect the invalid-name, zero-fan, and duplicate calls; explain why caller state is unchanged and identify the duplicate indices.
9. Check the Transformer paper’s initializer boundary and distinguish it from attention-score or embedding forward scaling.
10. Explain how the near-linear and independence assumptions limit the variance argument, distinguishing a target weight-distribution variance from a guarantee that every decoder signal retains its variance.

The important misconception is equating a chosen starting distribution with one matrix’s exact measured variance, or with a promise that activations can never vanish or grow. The sampler provides an explicit reproducible baseline; those stronger claims need separate evidence.

Give the initialized matrix token-lookup meaning

Initialization now supplies named trainable matrices with reproducible values and a declared starting scale. Chapter 18 turns a vocabulary-by-feature matrix into an embedding table: token IDs select rows, repeated occurrences of an ID reuse its row, and their gradients scatter-add. Passing vocabulary size and feature width to this sampler is our table-construction convention, not a variance derivation for row lookup.

One table is reused across batch items and sequence positions. That shared parameter identity is separate from the row selected by an individual token ID: different IDs select different rows, while repeated IDs share a row.

Keep construction order deliberate when several parameters share the initialization stream. Chapter 21 gives data shuffling a separately owned stream, Chapter 22 introduces optimizer state, and Chapter 35 persists parameter values and training provenance. The capability added here is reproducible construction of trainable leaves; token lookup and parameter updates are later operations.