17 · Content revision 4

Initialize trainable weights reproducibly

Initialize model weight matrices reproducibly, compare zero, oversized, and Xavier scales, and track expected variance through stacked linear layers.

Predict one seed, scale, and symmetry failure

Initialize a projection of shape [2,2] with seed 17, fan-in $2$ , and fan-out $2$ . Before running the example, predict a target variance of $1/2$ , a standard deviation of $1/\sqrt{2}$ , and a zero-centered uniform bound of $\sqrt{3/2}$ . The seeded generator produces these row-major values after twelve-decimal rounding:

 0.004950883736  -0.265932089217
-0.420504358848  -0.676313443233

The same seed, shape, fan values, and construction order must reproduce the same tensor bit for bit. For this selected request, alternate seed 18 produces a different tensor. Those relationships are testable even though the learner does not calculate the pseudorandom sequence by hand.

Now contrast the selected matrix’s unequal columns with a zero-initialized two-unit path. Let $x=[1,-1]$ , set a [2,2] input matrix to zero, apply SiLU, and use equal output weights [1,1]. The scalar output is zero. SiLU’s derivative at zero is $1/2$ , so both input-weight gradient columns are $[0.5,-0.5]^\top$ . An equal update therefore preserves the hidden units’ equality.

Differentiate the fixed zero-weight SiLU path and expose its equal gradient columns rust/demos/ch17-parameter-initialization/src/lib.rs#zero-symmetry-probe

pub fn zero_symmetry_probe() -> Result<SymmetryProbe, Box<dyn Error>> {
    let input = TensorValue::constant(tensor(&[1, 2], &[1.0, -1.0])?)?;
    let input_weights = TensorValue::parameter(tensor(&[2, 2], &[0.0; 4])?)?;
    let output_weights = TensorValue::constant(tensor(&[2, 1], &[1.0, 1.0])?)?;

    let hidden = input.matmul(&input_weights)?.silu()?;
    let output = hidden.matmul(&output_weights)?;
    let seed = tensor(&[1, 1], &[1.0])?;
    output.backward_with_seed(&seed.view(), GraphRetention::Retain)?;

    let output_value = output.value().as_slice()[0];
    let gradient = input_weights
        .gradient_snapshot()
        .expect("the input weights are a trainable leaf");
    let columns_equal = gradient.as_slice()[0] == gradient.as_slice()[1]
        && gradient.as_slice()[2] == gradient.as_slice()[3];
    Ok(SymmetryProbe {
        output: output_value,
        gradient,
        columns_equal,
    })
}

This is a bounded symmetry example, not a rule that every zero value is wrong. Zero biases and unit normalization gains can be deliberate because they do not create two equally treated trainable feature columns.

Target a distribution, not one exact finite sample

The chapter’s display formula is:

\operatorname{Var}(W_{ij})=\frac{2}{\operatorname{fan}_{in}+\operatorname{fan}_{out}}

For a zero-centered uniform distribution over $[-a,a)$ , the distribution variance is $a^2/3$ . Matching the display formula gives $a=\sqrt{6/(\operatorname{fan}_{in}+\operatorname{fan}_{out})}$ .

The word target matters. Four sampled numbers do not have to possess exactly the distribution’s theoretical variance. The rule also does not guarantee that signals cannot shrink or grow: its derivation uses simplifying independence and near-linear assumptions, while the eventual decoder includes nonlinearities, normalization, residual paths, data, and optimization.

More precisely, the balance assumes independent dense weights, input features with a common variance, and a symmetric activation near a linear unit-slope regime. SiLU does not exactly meet that activation assumption. In the decoder built here, learned matrices use the Xavier-style rule, optional biases start at zero, and RMSNorm gains start at one. The token table later reuses the matrix sampler with vocabulary size and feature width as its two shape inputs; that is an explicit convention, not a variance result derived from row lookup.

Name the weight and both widths

Symbol	Operational meaning
$W$	One weight matrix before training.
$i$	The input-coordinate index of one weight.
$j$	The output-coordinate index of one weight.
$W_{ij}$	The weight connecting input coordinate $i$ to output coordinate $j$ .
$\operatorname{Var}(W_{ij})$	The target variance of the initialization distribution, not one finite matrix’s measured variance.
$\operatorname{fan}_{in}$	The number of input values accumulated by one output.
$\operatorname{fan}_{out}$	The number of outputs receiving each input.
$2$	The compromise between the forward fan-in and backward fan-out variance conditions.

For fan-in $2$ and fan-out $2$ , the target standard deviation is $0.707106781187$ and the uniform bound is $1.224744871392$ . Doubling fan-in to $4$ while keeping fan-out $2$ reduces them to $0.577350269190$ and $1.000000000000$ . Width is part of the initialization request, not metadata that can be guessed after sampling.

From neural word features to width-aware decoder parameters

Bengio et al. jointly learn word features and neural matrices for next-word prediction and report random word-feature initialization similar to neural-network weight initialization. Their paper does not define a dimension-aware or reproducible initialization rule; arbitrary scales become more consequential when learned transformations are composed through depth.

The early neural-language-model checkpoint is Bengio et al., A Neural Probabilistic Language Model. Bengio et al. define a learned word-feature matrix and neural parameter matrices for next-word prediction, optimize them jointly, and report random initialization of the word features similarly to neural-network weights.

That source supports random starting values for learned language-model features and matrices. It does not specify a dimension-aware scale, exact distribution, seed, generator, stable names, or validation order.

Glorot and Bengio derive a normalized variance compromise for deep feed-forward networks under explicit near-linear and independence assumptions. Vaswani et al. later place learned embeddings at model boundaries and repeat attention projections, output projections, and feed-forward matrices through Transformer layers, making many width-dependent trainable matrices part of one language model.

The width-aware checkpoint is Glorot and Bengio, Understanding the difficulty of training deep feedforward neural networks. Glorot and Bengio balance fan-in and fan-out variance conditions under stated simplifying assumptions, yielding target variance 2 divided by their sum and a normalized zero-centered uniform initialization. Those assumptions motivate the scale; they do not exactly describe a SiLU, RMSNorm, and residual decoder.

The repeated learned-parameter checkpoint is Vaswani et al., Attention Is All You Need. Vaswani et al. use learned embeddings at the model boundaries and repeat query/key/value, attention-output, and two feed-forward projections in Transformer layers; the paper does not prescribe a parameter initializer. Its attention-score and embedding scaling are forward computations, not evidence for Xavier initialization.

This chapter gives later decoder weight matrices reproducible sampled values, stable names, and declared width-aware target variances. The decoder built here uses Xavier-style uniform matrix weights, zero optional biases, unit RMSNorm gains, and a shape-based token-table convention; these are explicit implementation policies, not claims that the original Transformer prescribed them or that every signal preserves variance exactly.

These choices connect learned neural word features to variance-aware deep transformations and then to repeated Transformer projections.

Generate and name parameters transactionally

The implementation uses the compact, fully specified SplitMix64 algorithm without an external dependency. The seed is the raw state before the first increment; zero is valid. Each draw maps the high 53 mixed bits to binary64 before the affine uniform transform. This is deterministic pseudorandom sampling, not cryptographic randomness.

Declare stable failures for invalid names, widths, shapes, allocation, tensors, and duplicates rust/crates/llm-from-scratch/src/nn/init.rs#parameter-init-errors

/// A deterministic rejection while constructing or collecting a parameter.
#[derive(Clone, Debug, PartialEq)]
pub enum InitializationError {
    EmptyName,
    EmptyNameSegment {
        index: usize,
    },
    InvalidNameCharacter {
        index: usize,
        byte: u8,
    },
    ZeroFanIn,
    ZeroFanOut,
    FanSumOverflow {
        fan_in: usize,
        fan_out: usize,
    },
    ShapeProductOverflow {
        fan_in: usize,
        fan_out: usize,
    },
    AllocationFailed {
        elements: usize,
    },
    DuplicateName {
        name: String,
        first: usize,
        repeated: usize,
    },
    Tensor(TensorError),
    Autodiff(TensorAutodiffError),
}

impl fmt::Display for InitializationError {
    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::EmptyName => formatter.write_str("parameter name must not be empty"),
            Self::EmptyNameSegment { index } => write!(
                formatter,
                "parameter name has an empty dot-separated segment at byte {index}"
            ),
            Self::InvalidNameCharacter { index, byte } => write!(
                formatter,
                "parameter name byte {index} must be lowercase ASCII, a digit, underscore, or dot; got 0x{byte:02x}"
            ),
            Self::ZeroFanIn => formatter.write_str("fan-in must be greater than zero"),
            Self::ZeroFanOut => formatter.write_str("fan-out must be greater than zero"),
            Self::FanSumOverflow { fan_in, fan_out } => write!(
                formatter,
                "fan-in {fan_in} plus fan-out {fan_out} does not fit usize"
            ),
            Self::ShapeProductOverflow { fan_in, fan_out } => write!(
                formatter,
                "matrix shape [{fan_in},{fan_out}] does not fit usize"
            ),
            Self::AllocationFailed { elements } => write!(
                formatter,
                "could not reserve storage for {elements} initialized values"
            ),
            Self::DuplicateName {
                name,
                first,
                repeated,
            } => write!(
                formatter,
                "parameter name {name:?} first appears at index {first} and repeats at index {repeated}"
            ),
            Self::Tensor(error) => error.fmt(formatter),
            Self::Autodiff(error) => error.fmt(formatter),
        }
    }
}

impl Error for InitializationError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        match self {
            Self::Tensor(error) => Some(error),
            Self::Autodiff(error) => Some(error),
            _ => None,
        }
    }
}

impl From<TensorError> for InitializationError {
    fn from(error: TensorError) -> Self {
        Self::Tensor(error)
    }
}

impl From<TensorAutodiffError> for InitializationError {
    fn from(error: TensorAutodiffError) -> Self {
        Self::Autodiff(error)
    }
}

Advance SplitMix64 and map its high 53 bits into one binary64 unit draw rust/crates/llm-from-scratch/src/nn/init.rs#deterministic-prng

/// A small deterministic generator with an explicit resumable 64-bit state.
///
/// This generator is suitable for reproducible teaching fixtures. It is not a
/// cryptographically secure random-number generator.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SplitMix64 {
    state: u64,
}

impl SplitMix64 {
    /// Treats `seed` as the raw state before the first increment and draw.
    pub const fn from_seed(seed: u64) -> Self {
        Self { state: seed }
    }

    /// Resumes directly from a previously recorded raw state.
    pub const fn from_state(state: u64) -> Self {
        Self { state }
    }

    /// Returns the raw state that the next draw will advance.
    pub const fn state(&self) -> u64 {
        self.state
    }

    /// Advances and mixes one exactly specified 64-bit value.
    pub fn next_u64(&mut self) -> u64 {
        self.state = self.state.wrapping_add(SPLITMIX_INCREMENT);
        let mut value = self.state;
        value = (value ^ (value >> 30)).wrapping_mul(SPLITMIX_MIX_ONE);
        value = (value ^ (value >> 27)).wrapping_mul(SPLITMIX_MIX_TWO);
        value ^ (value >> 31)
    }

    /// Maps the high 53 bits of one draw to the binary64 interval [0,1).
    pub fn next_unit_f64(&mut self) -> f64 {
        ((self.next_u64() >> 11) as f64) * BINARY64_UNIT_SCALE
    }
}

Validate both widths, derive the Xavier-style scale, and sample row-major values rust/crates/llm-from-scratch/src/nn/init.rs#xavier-initialization

/// The formula-derived scale for one [fan-in, fan-out] matrix.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct XavierScale {
    fan_in: usize,
    fan_out: usize,
    target_variance: f64,
    standard_deviation: f64,
    uniform_limit: f64,
}

impl XavierScale {
    pub const fn fan_in(self) -> usize {
        self.fan_in
    }

    pub const fn fan_out(self) -> usize {
        self.fan_out
    }

    pub const fn target_variance(self) -> f64 {
        self.target_variance
    }

    pub const fn standard_deviation(self) -> f64 {
        self.standard_deviation
    }

    pub const fn uniform_limit(self) -> f64 {
        self.uniform_limit
    }
}

/// Calculates the Xavier variance, standard deviation, and uniform bound.
pub fn xavier_scale(fan_in: usize, fan_out: usize) -> Result<XavierScale, InitializationError> {
    if fan_in == 0 {
        return Err(InitializationError::ZeroFanIn);
    }
    if fan_out == 0 {
        return Err(InitializationError::ZeroFanOut);
    }
    let fan_sum = fan_in
        .checked_add(fan_out)
        .ok_or(InitializationError::FanSumOverflow { fan_in, fan_out })?;
    let target_variance = 2.0 / fan_sum as f64;
    Ok(XavierScale {
        fan_in,
        fan_out,
        target_variance,
        standard_deviation: target_variance.sqrt(),
        uniform_limit: (6.0 / fan_sum as f64).sqrt(),
    })
}

fn checked_element_count(fan_in: usize, fan_out: usize) -> Result<usize, InitializationError> {
    fan_in
        .checked_mul(fan_out)
        .ok_or(InitializationError::ShapeProductOverflow { fan_in, fan_out })
}

pub(crate) fn validate_name(name: &str) -> Result<(), InitializationError> {
    if name.is_empty() {
        return Err(InitializationError::EmptyName);
    }

    let mut previous_was_dot = true;
    for (index, byte) in name.bytes().enumerate() {
        if byte == b'.' {
            if previous_was_dot {
                return Err(InitializationError::EmptyNameSegment { index });
            }
            previous_was_dot = true;
            continue;
        }

        if !(byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'_') {
            return Err(InitializationError::InvalidNameCharacter { index, byte });
        }
        previous_was_dot = false;
    }

    if previous_was_dot {
        return Err(InitializationError::EmptyNameSegment { index: name.len() });
    }
    Ok(())
}

fn initialized_values(
    rng: &mut SplitMix64,
    scale: XavierScale,
) -> Result<Vec<f64>, InitializationError> {
    let elements = checked_element_count(scale.fan_in, scale.fan_out)?;
    let mut values = Vec::new();
    values
        .try_reserve_exact(elements)
        .map_err(|_| InitializationError::AllocationFailed { elements })?;
    for _ in 0..elements {
        let centered = 2.0 * rng.next_unit_f64() - 1.0;
        values.push(scale.uniform_limit * centered);
    }
    Ok(values)
}

Validate immutable dot-separated names and preserve declaration order rust/crates/llm-from-scratch/src/nn/init.rs#named-parameters

/// One immutable external name paired with one trainable tensor-tape leaf.
#[derive(Clone, Debug)]
pub struct NamedParameter {
    name: String,
    tensor: TensorValue,
}

impl NamedParameter {
    /// Wraps an already-created tensor as a named trainable leaf.
    pub fn from_tensor(
        name: impl Into<String>,
        tensor: Tensor,
    ) -> Result<Self, InitializationError> {
        let name = name.into();
        validate_name(&name)?;
        Ok(Self {
            name,
            tensor: TensorValue::parameter(tensor)?,
        })
    }

    /// Samples one [fan-in, fan-out] trainable matrix transactionally.
    pub fn xavier_uniform(
        name: impl Into<String>,
        fan_in: usize,
        fan_out: usize,
        rng: &mut SplitMix64,
    ) -> Result<Self, InitializationError> {
        let name = name.into();
        validate_name(&name)?;
        let scale = xavier_scale(fan_in, fan_out)?;
        let elements = checked_element_count(fan_in, fan_out)?;

        let mut trial = rng.clone();
        let values = initialized_values(&mut trial, scale)?;
        debug_assert_eq!(values.len(), elements);
        let tensor = Tensor::from_vec(vec![fan_in, fan_out], values)?;
        let parameter = Self {
            name,
            tensor: TensorValue::parameter(tensor)?,
        };
        *rng = trial;
        Ok(parameter)
    }

    /// Returns the stable external identity used by layers and checkpoints.
    pub fn name(&self) -> &str {
        &self.name
    }

    /// Borrows the trainable tape leaf without duplicating its tensor storage.
    pub fn tensor(&self) -> &TensorValue {
        &self.tensor
    }
}

/// A duplicate-checked, declaration-ordered set of named parameters.
#[derive(Clone, Debug, Default)]
pub struct NamedParameters {
    parameters: Vec<NamedParameter>,
}

impl NamedParameters {
    pub fn try_new(parameters: Vec<NamedParameter>) -> Result<Self, InitializationError> {
        for repeated in 0..parameters.len() {
            if let Some(first) = parameters[..repeated]
                .iter()
                .position(|parameter| parameter.name() == parameters[repeated].name())
            {
                return Err(InitializationError::DuplicateName {
                    name: parameters[repeated].name().to_owned(),
                    first,
                    repeated,
                });
            }
        }
        Ok(Self { parameters })
    }

    pub fn len(&self) -> usize {
        self.parameters.len()
    }

    pub fn is_empty(&self) -> bool {
        self.parameters.is_empty()
    }

    pub fn as_slice(&self) -> &[NamedParameter] {
        &self.parameters
    }

    pub fn iter(&self) -> impl ExactSizeIterator<Item = &NamedParameter> {
        self.parameters.iter()
    }

    pub fn get(&self, name: &str) -> Option<&NamedParameter> {
        self.parameters
            .iter()
            .find(|parameter| parameter.name() == name)
    }
}

Names use nonempty lowercase ASCII segments separated by dots; each segment may also contain digits and underscores. Validation checks the name, zero fan-in, zero fan-out, fan-sum overflow, shape-product overflow, allocation, and tensor construction in declared order. Sampling uses a cloned trial generator and commits its state only after the trainable leaf exists, so every returned error leaves the caller’s generator unchanged.

A name is stable external identity. A cloned NamedParameter retains the same runtime tape node, while an independently recreated equal tensor is a different node. No generated numeric parameter IDs, layer abstractions, optimizer groups, or checkpoint format are introduced here.

Construct the selected projection and compare seed 17 with itself and seed 18 rust/demos/ch17-parameter-initialization/src/lib.rs#fixed-seed-parameter

    let scale = xavier_scale(2, 2)?;
    let mut rng = SplitMix64::from_seed(FIXTURE_SEED);
    let projection = projection_parameter(&mut rng)?;
    let weights = projection.tensor().value_snapshot();

    let mut matching_rng = SplitMix64::from_seed(FIXTURE_SEED);
    let matching = projection_parameter(&mut matching_rng)?;
    let mut alternate_rng = SplitMix64::from_seed(ALTERNATE_SEED);
    let alternate = projection_parameter(&mut alternate_rng)?;

Preserve declaration order and distinguish stable names from runtime node identity rust/demos/ch17-parameter-initialization/src/lib.rs#named-parameter-enumeration

    let token_table = NamedParameter::xavier_uniform("token_embedding.weight", 4, 2, &mut rng)?;
    let parameters = NamedParameters::try_new(vec![projection.clone(), token_table])?;

    let projection_clone = projection.clone();
    let clone_same_node = projection.tensor().is_same_node(projection_clone.tensor());
    let recreated_same_node = projection.tensor().is_same_node(matching.tensor());

Reject invalid requests without advancing the caller's generator rust/demos/ch17-parameter-initialization/src/lib.rs#initialization-errors-example

    let state_before_errors = rng.state();
    let invalid_name =
        NamedParameter::xavier_uniform("Decoder.weight", 2, 2, &mut rng).unwrap_err();
    let zero_fan_in =
        NamedParameter::xavier_uniform("decoder.invalid.weight", 0, 2, &mut rng).unwrap_err();
    let duplicate_name =
        NamedParameters::try_new(vec![projection.clone(), projection.clone()]).unwrap_err();
    let rng_unchanged = rng.state() == state_before_errors;

Prepare the exact scale, samples, symmetry, names, identity, errors, and handoff evidence rust/demos/ch17-parameter-initialization/src/main.rs#learner-parameter-initialization-output

    let report = learner_report()?;

    println!("seed: 17");
    println!("projection: shape=2x2 fan_in=2 fan_out=2");
    println!("target variance: {}", fixed(report.scale.target_variance()));
    println!("uniform limit: {}", fixed(report.scale.uniform_limit()));
    println!(
        "weights: {}",
        report
            .weights
            .as_slice()
            .iter()
            .map(|value| fixed(*value))
            .collect::<Vec<_>>()
            .join(",")
    );
    println!("same seed reproduces: {}", report.same_seed_reproduces);
    println!("different seed differs: {}", report.different_seed_differs);
    println!(
        "zero symmetry: output={} columns-equal={} gradient={}",
        fixed(report.symmetry.output),
        report.symmetry.columns_equal,
        report
            .symmetry
            .gradient
            .as_slice()
            .iter()
            .map(|value| fixed(*value))
            .collect::<Vec<_>>()
            .join(",")
    );
    println!(
        "parameters: {}",
        report
            .parameters
            .iter()
            .map(|parameter| format!(
                "{}[{}]",
                parameter.name(),
                shape(&parameter.tensor().shape())
            ))
            .collect::<Vec<_>>()
            .join(" | ")
    );
    println!(
        "identity: clone-same-node={} recreated-same-node={}",
        report.clone_same_node, report.recreated_same_node
    );
    println!(
        "validation: invalid-name | duplicate-name | zero-fan-in; rng-unchanged={}",
        report.rng_unchanged
    );
    println!("chapter 18 handoff: initialize a trainable token table");

Run the complete example from the repository root:

./course run cargo run --quiet --locked -p ch17-parameter-initialization

The output exposes the theoretical scale, the selected sampled values, the symmetry contrast, stable parameter names, and errors that leave the generator state unchanged.

Compare fixed-seed distributions and expected variance

The diagnostic expands the width to [64,64], giving 4096 weights per strategy. Zero uses no draws. Oversized uniform and Xavier-style uniform reuse the same seed-17 base draws, with the oversized bound exactly twice as large. That controlled pairing isolates scale while Rust records the histogram bins, counts, display percentages, and two-pass population statistics.

The second rail is assumption-bound. Starting from unit input variance, it records expected variance through four independent linear layers: zero collapses to zero, the doubled bound multiplies variance by four per layer, and the Xavier case stays at one. These are expected values under the declared linear independence assumptions, not measurements or guarantees for the nonlinear residual decoder.

Compute the paired distributions, histograms, theoretical propagation, and seed comparisons shown below rust/demos/ch17-parameter-initialization/src/diagram_trace.rs#parameter-initialization-trace

pub fn render_trace() -> Result<String, Box<dyn Error>> {
    let scale = xavier_scale(WIDTH, WIDTH)?;
    let mut rng = SplitMix64::from_seed(FIXTURE_SEED);
    let xavier = NamedParameter::xavier_uniform("diagnostic.weight", WIDTH, WIDTH, &mut rng)?
        .tensor()
        .value()
        .as_slice()
        .to_vec();
    let oversized: Vec<_> = xavier.iter().map(|value| value * 2.0).collect();
    let zero = vec![0.0; SAMPLE_COUNT];

    let zero_statistics = statistics(&zero);
    let oversized_statistics = statistics(&oversized);
    let xavier_statistics = statistics(&xavier);
    let zero_histogram = histogram(&zero);
    let oversized_histogram = histogram(&oversized);
    let xavier_histogram = histogram(&xavier);

    let mut same_seed_rng = SplitMix64::from_seed(FIXTURE_SEED);
    let same_seed =
        NamedParameter::xavier_uniform("diagnostic.weight", WIDTH, WIDTH, &mut same_seed_rng)?
            .tensor()
            .value()
            .as_slice()
            .to_vec();
    let mut alternate_rng = SplitMix64::from_seed(ALTERNATE_SEED);
    let alternate =
        NamedParameter::xavier_uniform("diagnostic.weight", WIDTH, WIDTH, &mut alternate_rng)?
            .tensor()
            .value()
            .as_slice()
            .to_vec();

    let mut trace = String::new();
    writeln!(trace, "TRACE parameter-initialization-v2 BEGIN")?;
    writeln!(
        trace,
        "FIXTURE name=fixed-seed-width64 generator=splitmix64 mapping=top53-affine seed=17 shape=64x64 samples=4096 fan-in=64 fan-out=64 statistic=population-two-pass layers=0,1,2,3,4 propagation=expected-linear-independent input-variance=1.000000000000 display-input-variance=1"
    )?;
    writeln!(
        trace,
        "BINNING edges={} display-edges={} width=0.100000000000 display-width=0.10 closure=left-closed-right-open-last-closed",
        fixed_list(&EDGES),
        DISPLAY_EDGES.join(",")
    )?;
    writeln!(
        trace,
        "{}",
        distribution_line("zero", "none", 0.0, &zero_statistics)
    )?;
    writeln!(trace, "{}", histogram_line("zero", &zero_histogram))?;
    writeln!(
        trace,
        "{}",
        distribution_line(
            "oversized",
            "17",
            scale.uniform_limit() * 2.0,
            &oversized_statistics,
        )
    )?;
    writeln!(
        trace,
        "{}",
        histogram_line("oversized", &oversized_histogram)
    )?;
    writeln!(
        trace,
        "{}",
        distribution_line("xavier", "17", scale.uniform_limit(), &xavier_statistics)
    )?;
    writeln!(trace, "{}", histogram_line("xavier", &xavier_histogram))?;
    writeln!(
        trace,
        "PAIRING seed=17 base-draws-equal=yes oversized-to-xavier-limit=2.000000000000"
    )?;
    writeln!(
        trace,
        "PROPAGATION kind=zero variances={} display-variances={}",
        fixed_list(&expected_variances(0.0)),
        display_integer_list(&expected_variances(0.0))
    )?;
    writeln!(
        trace,
        "PROPAGATION kind=oversized variances={} display-variances={}",
        fixed_list(&expected_variances(4.0)),
        display_integer_list(&expected_variances(4.0))
    )?;
    writeln!(
        trace,
        "PROPAGATION kind=xavier variances={} display-variances={}",
        fixed_list(&expected_variances(1.0)),
        display_integer_list(&expected_variances(1.0))
    )?;
    writeln!(
        trace,
        "REPRODUCIBILITY seed=17 same-seed-equal={} alternate-seed=18 alternate-seed-different={}",
        if xavier == same_seed { "yes" } else { "no" },
        if xavier != alternate { "yes" } else { "no" },
    )?;
    writeln!(trace, "TRACE parameter-initialization-v2 END")?;
    Ok(trace)
}

Compare zero weights with two paired scales

Compare measured finite-sample histograms for zero, oversized, and Xavier-style weights, then follow theoretical variance through four independent linear layers under the stated assumptions.

Shared seed: 17
Matrix shape: 64x64
Weight samples: 4096
Fan-in: 64
Fan-out: 64
Input variance: $1$
Generator and mapping: splitmix64 / top53-affine
Statistic: Population variance from a two-pass calculation

Compare fixed-seed weight distributions

All strategies share equal-width bins. Each interval includes its left edge; only the final interval includes its right edge. $\Delta=0.10$

All-zero weights

Seed: No draws
Uniform limit: 0.0000
Observed minimum: 0.0000
Observed maximum: 0.0000
Observed mean: 0.0000
Observed population variance: 0.0000

Double-width uniform weights

Seed: 17
Uniform limit: 0.4330
Observed minimum: -0.4330
Observed maximum: 0.4326
Observed mean: -0.0067
Observed population variance: 0.0632

Xavier-style uniform weights

Seed: 17
Uniform limit: 0.2165
Observed minimum: -0.2165
Observed maximum: 0.2163
Observed mean: -0.0034
Observed population variance: 0.0158

Controlled comparison

The two uniform samples use the same base draws.

Seed: 17
Oversized/Xavier bound ratio: 2.000000000000


0 0.0%	0 0.0%	0 0.0%	0 0.0%	4096 100.0%	0 0.0%	0 0.0%	0 0.0%	0 0.0%
409 10.0%	498 12.2%	482 11.8%	472 11.5%	469 11.5%	445 10.9%	476 11.6%	443 10.8%	402 9.8%
0 0.0%	0 0.0%	674 16.5%	962 23.5%	919 22.4%	930 22.7%	611 14.9%	0 0.0%	0 0.0%

Follow expected linear variance through depth

These are expected variances for independent linear layers with unit input variance. They are not measured guarantees for the nonlinear residual decoder.

Layer depth
0	$1$	$1$	$1$
1	$0$	$4$	$1$
2	$0$	$16$	$1$
3	$0$	$64$	$1$
4	$0$	$256$	$1$

Check what the seed does and does not fix

Same seed and request reproduce exactly

Seed: 17

The selected alternate seed differs

Seed: 18

The paired uniform samples reuse the same base draws, so their different bars and statistics come only from the doubled scale. The histogram variances are measurements from $4096$ values; the depth table instead follows the theoretical target-variance multipliers under the stated linear assumptions.

Predict before running Rust

For fan-in $2$ and fan-out $2$ , calculate the target variance, standard deviation, and uniform bound.
Recalculate those three quantities when fan-in becomes $4$ and fan-out remains $2$ .
Derive the two input-weight gradient columns for the zero-weight SiLU example.
Which values must match when seed, shape, fan values, and construction order all match?
Why does the selected seed 18 request differ without proving that every pair of possible seeds differs?
Why does one finite [2,2] tensor not need an empirical variance of exactly $1/2$ ?
In which order are decoder.block.0.attention.query.weight and token_embedding.weight enumerated?
Does a rejected invalid name or zero fan advance the generator, and which duplicate does a collection report?
Do Vaswani et al. prescribe Xavier initialization for the Transformer?
Does the formula guarantee exact realized variance or prevent every signal from shrinking or growing?

Check the predictions

The target variance is $0.5$ , the standard deviation is $0.707106781187$ , and the bound is $1.224744871392$ .
They become $1/3$ , $0.577350269190$ , and $1.000000000000$ .
Both columns are $[0.5,-0.5]^\top$ : the equal output weights and SiLU derivative $1/2$ give each hidden unit the same upstream factor.
The generator stream and resulting tensor bits match for this identical request.
The selected seed-18 request produces a different tensor in this example. A pseudorandom generator is deterministic, but that observation is not a mathematical uniqueness promise for every seed and request.
The formula describes a distribution. A small finite sample has sampling variation.
Declaration order is preserved: the query projection appears first, followed by the token table.
Neither rejection advances the generator. A collection reports the first repeated name together with its original and repeated indices.
No. The paper describes learned Transformer parameters but does not prescribe their initializer.
No. Xavier targets a sampling variance under simplifying assumptions; finite samples, nonlinearities, normalization, residual paths, depth, data, and optimization still affect propagation.

The central misconception is equating a target distribution variance with one matrix’s exact measured variance—or with a promise that activations can never vanish or explode. Xavier-style initialization is a clear bounded baseline, not such a guarantee.

Give initialization meaning as a token table

The cumulative implementation can now create named trainable matrices reproducibly at declared width-aware scales. Chapter 18 gives a token table embedding semantics: token IDs select rows, repeated IDs share one row, and their gradients scatter-add. Reusing the matrix sampler with vocabulary size and feature width as its two shape inputs is an explicit convention for that table, not a consequence of lookup variance.

Construction order matters when several parameters share one generator stream: inserting an earlier parameter changes all later draws. Keep that order deliberate and test it. Chapter 18 adds token lookup semantics, Chapter 21 keeps data shuffling on a separately owned stream, Chapter 22 introduces optimizer state, and Chapter 35 persists parameter values and training provenance.