33 · Content revision 10

Train every step, select with validation

Learn how a decoder training loop orders backpropagation, gradient clipping, scheduled AdamW updates, graph-free validation, and checkpoint selection without using test data.

Plan every update before you train

Run every step of a bounded decoder training plan, measure graph-free validation loss at fixed checkpoints, and restore the model state saved at the earliest checkpoint with minimum validation loss, all without consulting test data during this training execution. The worked fixture trains the one-block decoder from the previous chapter: vocabulary size $V=5$ , residual width $d_{\mathrm{model}}=4$ , two attention heads, feed-forward width $d_{\mathrm{ff}}=4$ , context length $T=2$ , and $144$ trainable scalars.

Train a deterministic one-block, 144-parameter decoder for eight fixed mini-batch updates with an explicit four-segment learning-rate schedule, global-norm clipping at $0.35$ , and validation measurements at steps $0,2,4,6,8$ . Before executing the first forward pass, the trainer already owns all of these decisions:

the update batches come only from Train and have fixed order;
every planned update has one finite positive learning rate;
each successful update uses forward>backward>finite-check>clip>adamw-step>zero-grad;
both reported losses are token-weighted means measured without a graph;
only Validation may select a state, while Test is rejected here.

The schedule is

(\eta_1,\ldots,\eta_8) =(0.040,0.040,0.025,0.025,0.015,0.015,0.008,0.008).

All eight steps execute. Validation does not stop the run early. In this fixture, the five validation losses are

(1.918167,1.696310,1.687788,1.642599,1.595297),

so the selected snapshot is $s^*=8$ . That happens to be the final planned state in this fixture; the selection rule itself does not prefer the last state.

Update on train, choose on validation

For update $s\in\{1,\ldots,8\}$ , the raw gradient is evaluated at the state before that update, and the parameter and moment states advance together:

\begin{aligned}g_s&=\nabla_\theta\mathcal{L}_{tr}^{(s)}(\theta_{s-1}),\\ \widetilde g_s&=\frac{c}{\max(c,\lVert g_s\rVert_2)}g_s,\\ (\theta_s,m_s,v_s)&=\operatorname{AdamW}_{\eta_s}\!\left(\theta_{s-1},\widetilde g_s,m_{s-1},v_{s-1}\right),\quad s=1,\ldots,8,\\ s^*&=\min\left\{s\in\mathcal{C}:\mathcal{L}_{va}(\theta_s)=\min_{k\in\mathcal{C}}\mathcal{L}_{va}(\theta_k)\right\}\end{aligned}

The two losses have different jobs. The gradient comes from $\mathcal{L}_{tr}$ and changes the parameters. The value $\mathcal{L}_{va}$ is measured only at the declared checkpoint steps and may replace the saved candidate, but it is never differentiated and never updates the decoder.

Treat $g_s$ as one conceptual vector formed by concatenating every coordinate $g_{s,p,i}$ from every named parameter $p$ . Here $\mathcal{P}$ is the set of all named trainable parameters, and $i$ indexes the scalar coordinates within parameter $p$ . This definition does not require an allocated concatenated tensor. The one global norm is

\lVert g_s\rVert_2= \sqrt{\sum_{p\in\mathcal{P}}\sum_i g_{s,p,i}^2}.

For ceiling $c=0.35$ , the trainer derives one scale from that complete norm and applies it to every coordinate:

\alpha_s=\frac{c}{\max(c,\lVert g_s\rVert_2)}, \qquad \widetilde g_s=\alpha_s g_s.

Because $c>0$ , the denominator never vanishes. A zero or already-small norm therefore uses $\alpha_s=1$ , while a larger norm is reduced to $c$ . At every declared schedule boundary, AdamW continues the existing moments and step counter while using the new rate $\eta_s$ ; changing the rate does not restart the optimizer.

For example, $\lVert g_s\rVert_2=1.4$ and $c=0.35$ give $\alpha_s=0.25$ . The trainer passes that one scalar with $\eta_s$ to AdamW. For every parameter $p$ and coordinate $i$ , the first-moment recurrence uses $\widetilde g_{s,p,i}=\alpha_sg_{s,p,i}$ , while the second-moment recurrence uses $\widetilde g_{s,p,i}^2=\alpha_s^2g_{s,p,i}^2$ . Thus the effective gradient has global norm $0.35$ , and the scale is applied before squaring. The raw gradient tensors remain unchanged on the existing parameter leaves until the trainer clears them after the update. The factor $\alpha_s$ does not scale AdamW’s separate weight-decay term.

Validation averages by predicted-token count, not by number of batches. If validation batch $j$ contains $n_j$ target tokens and has mean loss $\mathcal{L}^{(j)}_{va}$ , then

\mathcal{L}_{va} =\frac{\sum_j n_j\mathcal{L}^{(j)}_{va}}{\sum_j n_j}.

Selection ranges only over the measured checkpoint set $\mathcal{C}=\{0,2,4,6,8\}$ . On an exact tie, strict comparison retains the earlier candidate. Equivalently,

s^*=\min\left\{s\in\mathcal{C}:\mathcal{L}_{va}(\theta_s) =\min_{k\in\mathcal{C}}\mathcal{L}_{va}(\theta_k)\right\}.

Keep steps, gradients, and partitions distinct

$\theta_0$ is the initialized decoder; $\theta_s$ is its state after update $s$ .
$s\geq1$ is a one-based update index; checkpoint indices also include $0$ .
$\mathcal{L}_{tr}^{(s)}$ is next-token loss for training mini-batch $s$ .
$g_s=\nabla_\theta\mathcal{L}_{tr}^{(s)}(\theta_{s-1})$ conceptually concatenates every finite named-parameter gradient coordinate before clipping.
$\alpha_s$ is the single global clipping factor passed to AdamW; it is $1$ when no clipping is needed.
$\widetilde g_s$ is the globally clipped gradient AdamW uses to update both moments.
$c$ is the positive global-norm ceiling.
$\eta_s$ is the predetermined learning rate for update $s$ .
$m_s$ and $v_s$ are Adam’s continuing first- and second-moment states.
$\mathcal{L}_{va}$ is graph-free validation loss used only for selection.
$\mathcal{C}$ is the set of measured checkpoint indices.
$s^*$ is the earliest measured validation minimum.

The names Train, Validation, and Test are concrete Partition variants in the program. In the explanation, their roles are conceptual: training fits, validation selects, and test supplies evidence after selection. This trainer rejects Test throughout one training execution; Chapter 34’s later one-use count belongs to one local evaluator instance. A lower training loss cannot substitute for held-out validation evidence.

Parameter identity connects the registry to the computation. Each registry entry and its embedding, block, or normalization handle refer to the same TensorValue node. AdamW commits a new tensor value into that existing node, so all of those aliases observe the update without being rebuilt. The embedding lookup and output projection also continue to share one tied node. Replacing a registry entry with a different node would break this guarantee because the component handles would still refer to the old node.

From training-only reports to validation-selected LLM checkpoints

Move from fitting and reporting one training state toward predetermined mini-batch updates that produce periodic validation candidates while this trainer rejects the separate test partition throughout the training execution. These training practices form part of the road to modern LLMs.

A Neural Probabilistic Language Model provides an early example of this separation in a neural language model. Bengio and colleagues separate training, validation, and test text, explicitly associate validation with model selection and early stopping, and describe stochastic per-example parameter updates for a feed-forward neural language model. Full-corpus or per-example updates and training-set-only reporting do not by themselves define a scalable update cadence or an independent rule for choosing among candidate language-model states.

Sequence to Sequence Learning with Neural Networks adds practical sequence-training controls. Sutskever, Vinyals, and Le report batches of sequences, a predetermined learning-rate reduction policy, and rescaling when the global gradient norm crosses a fixed threshold in a recurrent sequence model.

Attention Is All You Need carries those ideas into the Transformer era. Vaswani and colleagues train Transformers with token-budgeted batches, a step-indexed warmup and inverse-square-root schedule, and periodically written checkpoints. Their reported checkpoint averaging is not the validation-minimum selection rule implemented here.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer explicitly describes validation-based checkpoint selection. Raffel and colleagues save fine-tuning checkpoints at a fixed cadence, choose the one with the best validation performance, and explicitly avoid using the test set for model selection.

Language Models are Few-Shot Learners shows how these controls are used in large-scale decoder-only training. Brown and colleagues carry Adam, scheduled learning rates, token-based batch scaling, and global-gradient-norm clipping into decoder-only language-model training at GPT-3 scale.

Neural language-model work separated train, validation, and test responsibilities; sequence and Transformer systems added mini-batches, explicit schedules, clipping, and periodic candidates; later text-to-text work stated that validation chooses a checkpoint so test data does not perform model selection. Decoder-only LLM training repeatedly forms token batches, differentiates the training objective, controls gradient magnitude, applies a step schedule, and measures held-out validation candidates while reserving test evidence for post-selection evaluation; this course’s one-use count belongs to one local evaluator instance. That count is not a repository-history claim.

These papers use different architectures and recipes. The course’s fixed seed, exact cadence, eight-step budget, and earliest-tie rule are local teaching choices, not universal properties of LLM training. The small runnable contrast shows why the selection signal matters: a training-only trace chooses its final entry, while a held-out validation trace can choose an earlier one.

Contrast selection by training loss with selection by a held-out validation trace rust/demos/ch33-training-selection/src/lib.rs#historical-selection-contrast

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct HistoricalSelection {
    pub training_only_step: usize,
    pub validation_step: usize,
}

fn earliest_minimum(values: &[f64]) -> usize {
    values
        .iter()
        .enumerate()
        .min_by(|left, right| left.1.total_cmp(right.1))
        .map_or(0, |(index, _)| index)
}

/// Contrasts a falling training trace with an earlier validation minimum.
pub fn historical_selection() -> HistoricalSelection {
    HistoricalSelection {
        training_only_step: earliest_minimum(&[2.0, 1.5, 1.2]),
        validation_step: earliest_minimum(&[2.0, 1.3, 1.4]),
    }
}

Represent the complete training plan explicitly in Rust

LearningRateSchedule stores exactly one finite positive rate per update. TrainerConfig requires validation at $s=0$ , strictly increasing checkpoint steps, a final checkpoint after the last update, and one positive finite global norm ceiling. The operation order is data, not an implication hidden inside a loop.

Validate the complete schedule, checkpoint cadence, clipping ceiling, and update order before execution rust/crates/llm-from-scratch/src/training/trainer.rs#training-plan

/// The exact learner-visible order of one successful parameter update.
pub const UPDATE_EVENT_ORDER: [&str; 6] = [
    "forward",
    "backward",
    "finite-check",
    "clip",
    "adamw-step",
    "zero-grad",
];

/// One finite positive learning rate for every planned update.
#[derive(Clone, Debug, PartialEq)]
pub struct LearningRateSchedule {
    rates: Vec<f64>,
}

impl LearningRateSchedule {
    pub fn new(rates: Vec<f64>) -> Result<Self, TrainerError> {
        if rates.is_empty() {
            return Err(TrainerError::EmptyLearningRateSchedule);
        }
        for (index, &value) in rates.iter().enumerate() {
            if !value.is_finite() || value <= 0.0 {
                return Err(TrainerError::InvalidScheduledLearningRate {
                    step: index + 1,
                    value,
                });
            }
        }
        Ok(Self { rates })
    }

    pub fn steps(&self) -> usize {
        self.rates.len()
    }

    pub fn learning_rate(&self, step: usize) -> Option<f64> {
        step.checked_sub(1)
            .and_then(|index| self.rates.get(index))
            .copied()
    }

    pub fn rates(&self) -> &[f64] {
        &self.rates
    }
}

/// Fixed update, validation, and clipping policy for one complete run.
#[derive(Clone, Debug, PartialEq)]
pub struct TrainerConfig {
    schedule: LearningRateSchedule,
    validation_steps: Vec<usize>,
    max_gradient_norm: f64,
}

impl TrainerConfig {
    pub fn new(
        schedule: LearningRateSchedule,
        validation_steps: Vec<usize>,
        max_gradient_norm: f64,
    ) -> Result<Self, TrainerError> {
        if !max_gradient_norm.is_finite() || max_gradient_norm <= 0.0 {
            return Err(TrainerError::InvalidMaximumGradientNorm {
                value: max_gradient_norm,
            });
        }
        if validation_steps.is_empty() {
            return Err(TrainerError::EmptyValidationSteps);
        }
        if validation_steps[0] != 0 {
            return Err(TrainerError::ValidationMustStartAtZero {
                actual: validation_steps[0],
            });
        }
        let final_step = schedule.steps();
        for (index, &step) in validation_steps.iter().enumerate() {
            if step > final_step {
                return Err(TrainerError::ValidationStepOutOfRange { step, final_step });
            }
            if index > 0 && step <= validation_steps[index - 1] {
                return Err(TrainerError::ValidationStepsNotIncreasing {
                    previous: validation_steps[index - 1],
                    next: step,
                });
            }
        }
        let actual = *validation_steps
            .last()
            .expect("a nonempty validation schedule has a last step");
        if actual != final_step {
            return Err(TrainerError::ValidationMustEndAtFinalStep {
                expected: final_step,
                actual,
            });
        }
        Ok(Self {
            schedule,
            validation_steps,
            max_gradient_norm,
        })
    }

    pub const fn schedule(&self) -> &LearningRateSchedule {
        &self.schedule
    }

    pub fn validation_steps(&self) -> &[usize] {
        &self.validation_steps
    }

    pub const fn max_gradient_norm(&self) -> f64 {
        self.max_gradient_norm
    }
}

Graph-free evaluation needs a real tape boundary. Calling value or detach after a forward pass would be too late because the intermediate graph would already exist. The thread-local no_grad scope suppresses parent edges as each operation is created, supports nesting, and restores recording during unwinding.

Suppress reverse-mode parent edges in a nestable and panic-safe scope rust/crates/llm-from-scratch/src/autograd/tensor_core.rs#no-grad-scope

thread_local! {
    static NO_GRAD_DEPTH: Cell<usize> = const { Cell::new(0) };
}

struct NoGradGuard;

impl Drop for NoGradGuard {
    fn drop(&mut self) {
        NO_GRAD_DEPTH.with(|depth| {
            depth.set(
                depth
                    .get()
                    .checked_sub(1)
                    .expect("a no-grad guard must balance one entered scope"),
            );
        });
    }
}

fn no_grad_active() -> bool {
    NO_GRAD_DEPTH.with(|depth| depth.get() != 0)
}

/// Runs `operation` without recording reverse-mode parent edges.
///
/// The scope is thread-local, nestable, and restored even if `operation`
/// unwinds. Forward arithmetic and finite-value checks are unchanged, but every
/// result created inside the scope is untracked and cannot mutate parameter
/// gradients through `backward`.
pub fn no_grad<T>(operation: impl FnOnce() -> T) -> T {
    NO_GRAD_DEPTH.with(|depth| {
        depth.set(
            depth
                .get()
                .checked_add(1)
                .expect("no-grad nesting depth must fit usize"),
        );
    });
    let _guard = NoGradGuard;
    operation()
}

evaluate_no_grad computes a token-weighted epoch mean and proves that every result is untracked. It snapshots gradient bits before evaluation and rejects the call if any bit changes.

Measure token-weighted loss without recording a graph or changing gradients rust/crates/llm-from-scratch/src/training/trainer.rs#no-grad-evaluation

/// Evaluates one epoch without recording parent edges or mutating gradients.
pub fn evaluate_no_grad(
    model: &DecoderModel,
    epoch: &MiniBatchEpoch,
) -> Result<Evaluation, TrainerError> {
    if epoch.window_count() == 0 {
        return Err(TrainerError::EmptyEpoch {
            role: if epoch.partition() == Partition::Validation {
                TrainerEpochRole::ValidationSelection
            } else {
                TrainerEpochRole::TrainEvaluation
            },
        });
    }
    let before = gradient_bits(model)?;
    let mut weighted_sum = 0.0;
    let mut token_count = 0_usize;
    let mut recorded_graphs = 0_usize;
    for (batch_index, batch) in epoch.batches().iter().enumerate() {
        let loss = no_grad(|| {
            model.loss(
                batch.inputs(),
                &[batch.batch_width(), batch.context_length()],
                batch.targets(),
            )
        })?;
        if loss.tracks_gradient() {
            return Err(TrainerError::ValidationRecordedGraph {
                partition: epoch.partition(),
                batch: batch_index,
            });
        }
        recorded_graphs += usize::from(loss.tracks_gradient());
        let scalar = scalar_loss(&loss)?;
        weighted_sum += scalar * batch.token_count() as f64;
        token_count += batch.token_count();
    }
    let mean_loss = weighted_sum / token_count as f64;
    if !mean_loss.is_finite() {
        return Err(TrainerError::NonFiniteLoss { value: mean_loss });
    }
    if gradient_bits(model)? != before {
        return Err(TrainerError::ValidationChangedGradient);
    }
    Ok(Evaluation {
        mean_loss,
        token_count,
        batch_count: epoch.batch_count(),
        recorded_graphs,
    })
}

Before the first update, train_decoder must isolate its working decoder from the model borrowed from its caller. DecoderModelState::snapshot therefore copies every named parameter tensor once into graph-free state. The trainer immediately calls into_model on that owned state: each name and tensor buffer moves into the working decoder instead of being copied a second time. The trainer also clones the caller’s optimizer once. That working decoder and optimizer then persist through all eight updates, so neither is reconstructed for each mini-batch.

Later snapshots solve a different problem. When a validation checkpoint becomes the new minimum, its parameter values must remain unchanged while later updates continue to mutate the working decoder, so that minimum requires a deep snapshot. At the end, TrainingResult keeps the selected graph-free state as the immutable record of what validation chose and a separate decoder with the same values for later evaluation. Because both must remain available, restore_independent_model makes the one additional buffer copy.

Copy buffers when two results must remain independently usable, then move buffers when owned state is consumed rust/crates/llm-from-scratch/src/training/trainer.rs#decoder-state-snapshot

#[derive(Clone, Debug, PartialEq)]
struct StateParameter {
    name: String,
    value: Tensor,
}

/// Graph-free owned values for one complete decoder state.
#[derive(Debug, PartialEq)]
pub struct DecoderModelState {
    config: DecoderModelConfig,
    parameters: Vec<StateParameter>,
}

impl DecoderModelState {
    /// Copies a live decoder into graph-free state that can outlive later updates.
    pub fn snapshot(model: &DecoderModel) -> Self {
        Self {
            config: model.config(),
            parameters: model
                .parameters()
                .iter()
                .map(|parameter| StateParameter {
                    name: parameter.name().to_owned(),
                    value: parameter.tensor().value_snapshot(),
                })
                .collect(),
        }
    }

    /// Copies this state when two independent owners must retain the same values.
    pub fn independent_snapshot(&self) -> Self {
        Self {
            config: self.config,
            parameters: self.parameters.clone(),
        }
    }

    pub const fn config(&self) -> DecoderModelConfig {
        self.config
    }

    pub fn parameter_names(&self) -> impl ExactSizeIterator<Item = &str> {
        self.parameters
            .iter()
            .map(|parameter| parameter.name.as_str())
    }

    pub fn scalar_count(&self) -> usize {
        self.parameters
            .iter()
            .map(|parameter| parameter.value.len())
            .sum()
    }

    pub fn bit_pattern(&self) -> Vec<u64> {
        self.parameters
            .iter()
            .flat_map(|parameter| {
                parameter
                    .value
                    .as_slice()
                    .iter()
                    .map(|value| value.to_bits())
            })
            .collect()
    }

    /// Rebuilds an independent decoder while retaining this state snapshot.
    pub fn restore_independent_model(&self) -> Result<DecoderModel, TrainerError> {
        self.independent_snapshot().into_model()
    }

    /// Consumes graph-free state and moves every tensor buffer into one decoder.
    pub fn into_model(self) -> Result<DecoderModel, TrainerError> {
        let Self { config, parameters } = self;
        let parameters = parameters
            .into_iter()
            .map(|parameter| NamedParameter::from_tensor(parameter.name, parameter.value))
            .collect::<Result<Vec<_>, _>>()?;
        DecoderModel::from_parameters(config, parameters).map_err(Into::into)
    }
}

/// One validation-selected model and its AdamW state captured at the same step.
///
/// Only the trainer can construct this bundle. Callers may inspect it, but they
/// cannot attach a freely supplied step label or optimizer from another point in
/// the run before passing it to a checkpoint.
///
/// ```compile_fail
/// use llm_from_scratch::training::trainer::SelectedTrainingState;
///
/// let _counterfeit = SelectedTrainingState {
///     step: 8,
///     model_state: todo!(),
///     optimizer_state: todo!(),
/// };
/// ```
#[derive(Debug, PartialEq)]
pub struct SelectedTrainingState {
    step: usize,
    model_state: DecoderModelState,
    optimizer_state: AdamWState,
}

impl SelectedTrainingState {
    pub const fn step(&self) -> usize {
        self.step
    }

    pub const fn model_state(&self) -> &DecoderModelState {
        &self.model_state
    }

    pub const fn optimizer_state(&self) -> &AdamWState {
        &self.optimizer_state
    }
}

After checking every raw gradient and computing one norm over all named coordinates, the trainer derives one $\alpha_s$ . It passes the working model’s existing parameter handles, $\eta_s$ , and $\alpha_s$ to the same working optimizer’s step_with_learning_rate_and_gradient_scale method. The method returns only the committed optimizer step number; Chapter 22’s more detailed trace is unnecessary inside this loop.

AdamW first validates the complete prospective update. Its first moments use $\widetilde g_{s,p,i}$ and its second moments use $\widetilde g_{s,p,i}^2$ , so clipping is applied before the square. Decoupled weight decay still uses the old parameter value and is not multiplied by $\alpha_s$ . AdamW prepares every next parameter tensor, both moment states, and the step number before acquiring write access to all parameter values. A failure during preparation or write acquisition leaves every parameter and optimizer field unchanged. AdamW does hold each fully checked prospective tensor value until the transaction can commit. Those prepared values are required optimizer transaction state, not a replacement parameter vector owned by the trainer.

After all checks succeed, AdamW writes each prepared tensor into its existing TensorValue node and commits the prepared optimizer state. Names, parameter order, node identity, and the embedding/output tie do not change. The registry and every decoder component already hold aliases of those nodes, so the next forward pass observes the new values without rebuilding the decoder.

AdamW deliberately leaves each raw gradient tensor unchanged on its parameter node. The trainer compares the returned optimizer step number with the planned update index, calls zero_grad() on every live parameter, and verifies that every gradient coordinate is zero before the next forward pass. This explicit trainer operation prevents accumulation across mini-batches. No candidate decoder, candidate optimizer, or replacement Vec<NamedParameter> is constructed by the trainer for each update.

Apply the scheduled rate and one validated clipping factor without retaining a trace rust/crates/llm-from-scratch/src/training/adamw.rs#adamw-execution-and-trace-api

    /// Reads the accumulated gradients and atomically updates every live leaf.
    ///
    /// All arithmetic, tensor construction, and optimizer-state changes are
    /// prepared first. An error leaves both the supplied parameters and this
    /// optimizer bit-identical. A successful commit preserves every parameter
    /// node and leaves its accumulated gradient for the caller to clear.
    /// The result is only the committed step number; use `step_with_trace` when
    /// the elementwise update vectors are needed for inspection.
    pub fn step(&mut self, parameters: &[NamedParameter]) -> Result<u64, AdamWError> {
        self.step_with_config(parameters, self.config, 1.0, NoAdamWTrace)
    }

    /// Applies the same transaction while recording every elementwise update.
    pub fn step_with_trace(
        &mut self,
        parameters: &[NamedParameter],
    ) -> Result<AdamWStep, AdamWError> {
        let observer = RecordAdamWTrace::with_capacity(parameters.len());
        self.step_with_config(parameters, self.config, 1.0, observer)
    }

    /// Applies one validated scheduled learning rate without resetting moments.
    ///
    /// The override belongs only to this update; `config()` keeps the optimizer's
    /// base rate. An invalid rate or any later preparation error leaves the
    /// parameters, moments, powers, and step counter unchanged.
    pub fn step_with_learning_rate(
        &mut self,
        parameters: &[NamedParameter],
        learning_rate: f64,
    ) -> Result<u64, AdamWError> {
        let step_config = self.config.with_learning_rate(learning_rate)?;
        self.step_with_config(parameters, step_config, 1.0, NoAdamWTrace)
    }

    /// Applies one scheduled rate and one validated global gradient scale.
    ///
    /// The scale multiplies only the gradient used by Adam's moments. The
    /// decoupled weight-decay branch continues to use the unscaled parameter.
    pub fn step_with_learning_rate_and_gradient_scale(
        &mut self,
        parameters: &[NamedParameter],
        learning_rate: f64,
        gradient_scale: f64,
    ) -> Result<u64, AdamWError> {
        let step_config = self.config.with_learning_rate(learning_rate)?;
        self.step_with_config(parameters, step_config, gradient_scale, NoAdamWTrace)
    }

    /// Applies a scheduled learning rate and records the complete update trace.
    pub fn step_with_learning_rate_and_trace(
        &mut self,
        parameters: &[NamedParameter],
        learning_rate: f64,
    ) -> Result<AdamWStep, AdamWError> {
        let step_config = self.config.with_learning_rate(learning_rate)?;
        let observer = RecordAdamWTrace::with_capacity(parameters.len());
        self.step_with_config(parameters, step_config, 1.0, observer)
    }

DecoderModel::from_parameters remains a reconstruction boundary, not an ordinary AdamW operation. into_model first moves each owned tensor buffer into one new parameter leaf. from_parameters then borrows that stable-order leaf list and applies the shared decoder-layout rules: valid configuration, exact $2+9N$ tensor count, one required name at every list index, and the shape required by each component. The layout check neither copies a tensor buffer nor creates component handles. Passing it means that the leaf list has a valid layout; it does not yet mean that live decoder aliases exist.

After validation, from_parameters gives the embedding, blocks, and final RMSNorm shared handles to those leaves. Only this binding step re-establishes one embedding node for both lookup and output projection. Cloning the handles does not copy their tensor buffers. Reconstruction occurs when owned state becomes a model; the trainer does not call this boundary after an ordinary AdamW step.

Bind moved parameter buffers into one decoder and re-establish every component alias rust/crates/llm-from-scratch/src/models/decoder.rs#decoder-parameter-rebuild

    /// Rebuilds every component handle from one exact stable-order parameter set.
    ///
    /// State restoration uses this construction boundary to create an isolated
    /// decoder. Ordinary optimizer steps instead update the existing leaves, so
    /// the registry, components, and tied embedding keep their live aliases.
    pub fn from_parameters(
        config: DecoderModelConfig,
        parameters: Vec<NamedParameter>,
    ) -> Result<Self, DecoderModelError> {
        validate_parameter_layout(config, parameters.as_slice())?;
        let expected = parameters.len();

        let embedding = Embedding::from_parameter(parameters[0].clone())
            .map_err(DecoderModelError::Embedding)?;
        let mut blocks = Vec::new();
        blocks.try_reserve_exact(config.layers).map_err(|_| {
            DecoderModelError::LayerAllocationFailed {
                layers: config.layers,
            }
        })?;
        for layer in 0..config.layers {
            let start = 1 + layer * BLOCK_PARAMETER_SUFFIXES.len();
            let attention_norm = RmsNorm::from_gain(parameters[start].clone(), config.rms_epsilon)
                .map_err(|source| DecoderModelError::Block {
                    layer,
                    source: DecoderBlockError::AttentionNorm(source),
                })?;
            let attention = MultiHeadAttention::from_parameters(
                parameters[start + 1].clone(),
                parameters[start + 2].clone(),
                parameters[start + 3].clone(),
                parameters[start + 4].clone(),
                config.heads,
                config.max_positions,
                config.rope_base,
            )
            .map_err(|source| DecoderModelError::Block {
                layer,
                source: DecoderBlockError::Attention(source),
            })?;
            let feed_forward_norm =
                RmsNorm::from_gain(parameters[start + 5].clone(), config.rms_epsilon).map_err(
                    |source| DecoderModelError::Block {
                        layer,
                        source: DecoderBlockError::FeedForwardNorm(source),
                    },
                )?;
            let feed_forward = SwiGlu::from_parameters(
                parameters[start + 6].clone(),
                parameters[start + 7].clone(),
                parameters[start + 8].clone(),
            )
            .map_err(|source| DecoderModelError::Block {
                layer,
                source: DecoderBlockError::FeedForward(source),
            })?;
            blocks.push(
                DecoderBlock::from_parts(
                    attention_norm,
                    attention,
                    feed_forward_norm,
                    feed_forward,
                )
                .map_err(|source| DecoderModelError::Block { layer, source })?,
            );
        }
        let final_norm = RmsNorm::from_gain(parameters[expected - 1].clone(), config.rms_epsilon)
            .map_err(DecoderModelError::FinalNorm)?;
        Self::from_parts(config, embedding, blocks, final_norm)
    }

train_decoder first rejects the wrong partitions and incompatible epochs. It then creates the one isolated working model, executes all scheduled updates, measures only declared checkpoints, and replaces the selected snapshot only on strict loss improvement. After the final update, it returns the retained selected snapshot together with its explicitly independent decoder. The caller’s model and optimizer remain unchanged even on failure.

Update one persistent decoder, snapshot strict validation improvements, and return the earliest minimum while this training execution rejects test data rust/crates/llm-from-scratch/src/training/trainer.rs#complete-training-loop

/// Executes every planned update, then restores the earliest validation minimum.
///
/// The supplied model and optimizer remain unchanged. Update batches may cycle
/// deterministically, but validation never stops the run or consults test data.
pub fn train_decoder(
    initial_model: &DecoderModel,
    initial_optimizer: &AdamW,
    update_epoch: &MiniBatchEpoch,
    train_evaluation: &MiniBatchEpoch,
    validation: &MiniBatchEpoch,
    config: &TrainerConfig,
) -> Result<TrainingResult, TrainerError> {
    if initial_optimizer.step_count() != 0 {
        return Err(TrainerError::OptimizerNotFresh {
            step: initial_optimizer.step_count(),
        });
    }
    let context_length = update_epoch.context_length();
    let model_config = initial_model.config();
    validate_epoch(
        TrainerEpochRole::Update,
        Partition::Train,
        context_length,
        model_config,
        update_epoch,
    )?;
    validate_epoch(
        TrainerEpochRole::TrainEvaluation,
        Partition::Train,
        context_length,
        model_config,
        train_evaluation,
    )?;
    validate_epoch(
        TrainerEpochRole::ValidationSelection,
        Partition::Validation,
        context_length,
        model_config,
        validation,
    )?;

    let model = DecoderModelState::snapshot(initial_model).into_model()?;
    let mut optimizer = initial_optimizer.clone();
    let mut checkpoints = vec![checkpoint(0, &model, train_evaluation, validation)?];
    let mut selected_step = 0_usize;
    let mut selected_validation_loss = checkpoints[0].validation.mean_loss();
    let mut selected_state = DecoderModelState::snapshot(&model);
    let mut selected_optimizer_state = optimizer.persistence_state();
    let mut steps = Vec::with_capacity(config.schedule.steps());

    for step in 1..=config.schedule.steps() {
        let batch_index = (step - 1) % update_epoch.batch_count();
        let batch = &update_epoch.batches()[batch_index];
        let loss = model.loss(
            batch.inputs(),
            &[batch.batch_width(), batch.context_length()],
            batch.targets(),
        )?;
        let train_loss = scalar_loss(&loss)?;
        loss.backward_with_seed(
            &Tensor::from_vec(Vec::new(), vec![1.0])?.view(),
            GraphRetention::Release,
        )?;
        drop(loss);

        let norm = gradient_norm(&model, config.max_gradient_norm)?;
        let learning_rate = config
            .schedule
            .learning_rate(step)
            .expect("the loop stays inside the validated schedule");
        let optimizer_step = optimizer.step_with_learning_rate_and_gradient_scale(
            model.parameters(),
            learning_rate,
            norm.scale,
        )?;
        let expected_optimizer_step = u64::try_from(step).unwrap_or(u64::MAX);
        if optimizer_step != expected_optimizer_step {
            return Err(TrainerError::OptimizerStepMismatch {
                expected: expected_optimizer_step,
                actual: optimizer_step,
            });
        }
        clear_and_verify_gradients(&model)?;

        let batch_windows = batch
            .provenance()
            .iter()
            .map(|window| format!("{}@{}", window.document_id(), window.start()))
            .collect();
        steps.push(TrainingStep {
            step,
            batch_windows,
            learning_rate,
            train_loss,
            gradient_norm_before: norm.before,
            gradient_norm_after: norm.after,
            gradient_scale: norm.scale,
            clipped: norm.scale < 1.0,
            finite_gradients: true,
            parameter_nodes_preserved: true,
            cleared_gradients: true,
            events: UPDATE_EVENT_ORDER,
        });

        if config.validation_steps.binary_search(&step).is_ok() {
            let measured = checkpoint(step, &model, train_evaluation, validation)?;
            if measured.validation.mean_loss() < selected_validation_loss {
                selected_step = step;
                selected_validation_loss = measured.validation.mean_loss();
                selected_state = DecoderModelState::snapshot(&model);
                selected_optimizer_state = optimizer.persistence_state();
            }
            checkpoints.push(measured);
        }
    }

    for measured in &mut checkpoints {
        measured.selected = measured.step == selected_step;
    }
    let final_state = DecoderModelState::snapshot(&model);
    let selected_model = selected_state.restore_independent_model()?;
    Ok(TrainingResult {
        steps,
        checkpoints,
        selected_validation_loss,
        selected_training_state: SelectedTrainingState {
            step: selected_step,
            model_state: selected_state,
            optimizer_state: selected_optimizer_state,
        },
        final_state,
        selected_model,
        final_optimizer: optimizer,
    })
}

The deterministic fixture runs the complete plan twice, checks bitwise replay, confirms clipping occurred, proves evaluation recorded no graphs, and verifies that the selected model reproduces the saved parameter bits.

The exact-state checks need an owned vector of bit patterns, not owned copies of the parameter tensors. parameter_bits temporarily reads each parameter tensor, converts its scalar values to u64, and retains only the resulting bit vector after the read ends.

Collect clipping, graph, ownership, selection, and replay evidence from two complete runs rust/demos/ch33-training-selection/src/lib.rs#learner-evidence

#[derive(Debug)]
struct PreparedEpochs {
    updates: MiniBatchEpoch,
    train_evaluation: MiniBatchEpoch,
    validation: MiniBatchEpoch,
    test_probe: MiniBatchEpoch,
}

fn epoch(
    partition: Partition,
    documents: &[(&str, &[u32])],
    order: BatchOrder,
    batch_size: usize,
) -> Result<MiniBatchEpoch, FixtureError> {
    let documents = documents
        .iter()
        .map(|(id, token_ids)| BatchDocument::new(id, partition, token_ids))
        .collect::<Result<Vec<_>, _>>()?;
    let windows = CausalWindowConfig::new(CONTEXT_LENGTH, 1)
        .map_err(|_| FixtureError::Invariant("fixed window configuration changed"))?;
    let batches = MiniBatchConfig::new(batch_size, order)?;
    MiniBatchEpoch::build(partition, &documents, windows, batches).map_err(Into::into)
}

fn prepared_epochs() -> Result<PreparedEpochs, FixtureError> {
    let train_documents = [
        ("train-a", TRAIN_A.as_slice()),
        ("train-b", TRAIN_B.as_slice()),
    ];
    let validation_documents = [
        ("validation-a", VALIDATION_A.as_slice()),
        ("validation-b", VALIDATION_B.as_slice()),
    ];
    let updates = epoch(
        Partition::Train,
        &train_documents,
        BatchOrder::Shuffled { seed: SHUFFLE_SEED },
        BATCH_SIZE,
    )?;
    let train_evaluation = epoch(
        Partition::Train,
        &train_documents,
        BatchOrder::Sequential,
        5,
    )?;
    let validation = epoch(
        Partition::Validation,
        &validation_documents,
        BatchOrder::Sequential,
        4,
    )?;
    let test_probe = epoch(
        Partition::Test,
        &validation_documents,
        BatchOrder::Sequential,
        4,
    )?;
    require(
        updates.window_count() == 20,
        "training window count changed",
    )?;
    require(updates.batch_count() == 10, "training batch count changed")?;
    require(
        train_evaluation.window_count() == 20,
        "training evaluation window count changed",
    )?;
    require(
        validation.window_count() == 14,
        "validation window count changed",
    )?;
    Ok(PreparedEpochs {
        updates,
        train_evaluation,
        validation,
        test_probe,
    })
}

pub fn fixture_model_config() -> DecoderModelConfig {
    DecoderModelConfig::new(
        VOCABULARY_SIZE,
        MODEL_WIDTH,
        HEADS,
        FEED_FORWARD_WIDTH,
        LAYERS,
        CONTEXT_LENGTH,
        10_000.0,
        1e-6,
    )
}

pub fn fixture_trainer_config() -> Result<TrainerConfig, FixtureError> {
    TrainerConfig::new(
        LearningRateSchedule::new(LEARNING_RATES.to_vec())?,
        VALIDATION_STEPS.to_vec(),
        MAX_GRADIENT_NORM,
    )
    .map_err(Into::into)
}

fn fixture_optimizer(model: &DecoderModel) -> Result<AdamW, FixtureError> {
    let mut decay = Vec::new();
    let mut no_decay = Vec::new();
    for parameter in model.parameters() {
        if parameter.name().ends_with(".gain") {
            no_decay.push(parameter.name().to_owned());
        } else {
            decay.push(parameter.name().to_owned());
        }
    }
    let groups = AdamWParameterGroups::new(decay, no_decay)?;
    let config = AdamWConfig::new(LEARNING_RATES[0], 0.9, 0.999, 1e-8, 0.01)?;
    Ok(AdamW::with_parameter_groups(config, groups))
}

#[derive(Debug)]
struct SingleRun {
    result: TrainingResult,
    input_model_unchanged: bool,
    input_optimizer_unchanged: bool,
    test_partition_rejected: bool,
}

fn parameter_bits(model: &DecoderModel) -> Vec<u64> {
    let mut bits = Vec::new();
    for parameter in model.parameters() {
        bits.extend(
            parameter
                .tensor()
                .value()
                .as_slice()
                .iter()
                .map(|value| value.to_bits()),
        );
    }
    bits
}

fn run_once() -> Result<SingleRun, FixtureError> {
    let epochs = prepared_epochs()?;
    let model = DecoderModel::new(
        fixture_model_config(),
        &mut SplitMix64::from_seed(INIT_SEED),
    )?;
    require(
        model.parameters().len() == 11,
        "parameter tensor count changed",
    )?;
    require(
        model.parameter_count() == 144,
        "parameter scalar count changed",
    )?;
    let initial_bits = parameter_bits(&model);
    let optimizer = fixture_optimizer(&model)?;
    let result = train_decoder(
        &model,
        &optimizer,
        &epochs.updates,
        &epochs.train_evaluation,
        &epochs.validation,
        &fixture_trainer_config()?,
    )?;
    let test_partition_rejected = matches!(
        train_decoder(
            &model,
            &optimizer,
            &epochs.test_probe,
            &epochs.train_evaluation,
            &epochs.validation,
            &fixture_trainer_config()?,
        ),
        Err(TrainerError::WrongPartition {
            role: llm_from_scratch::training::trainer::TrainerEpochRole::Update,
            expected: Partition::Train,
            actual: Partition::Test,
        })
    );
    let model_after = parameter_bits(&model);
    Ok(SingleRun {
        result,
        input_model_unchanged: model_after == initial_bits,
        input_optimizer_unchanged: optimizer.step_count() == 0
            && optimizer.parameter_names().next().is_none(),
        test_partition_rejected,
    })
}

fn replay_equal(left: &SingleRun, right: &SingleRun) -> bool {
    left.result.steps() == right.result.steps()
        && left.result.checkpoints() == right.result.checkpoints()
        && left.result.selected_step() == right.result.selected_step()
        && left.result.selected_validation_loss().to_bits()
            == right.result.selected_validation_loss().to_bits()
        && left.result.selected_state().bit_pattern() == right.result.selected_state().bit_pattern()
        && left.result.final_state().bit_pattern() == right.result.final_state().bit_pattern()
}

#[derive(Debug)]
pub struct LearnerEvidence {
    pub result: TrainingResult,
    pub replay_bitwise: bool,
    pub input_model_unchanged: bool,
    pub input_optimizer_unchanged: bool,
    pub test_partition_rejected: bool,
    /// Compatibility evidence for the final-evaluation boundary: rejection
    /// before a forward pass implies that selection consumed zero test epochs.
    pub test_reads: usize,
    pub history: HistoricalSelection,
}

pub fn learner_evidence() -> Result<LearnerEvidence, FixtureError> {
    let first = run_once()?;
    let second = run_once()?;
    require(
        first.result.steps().len() == LEARNING_RATES.len(),
        "step count changed",
    )?;
    require(
        first.result.checkpoints().len() == VALIDATION_STEPS.len(),
        "checkpoint count changed",
    )?;
    require(
        first
            .result
            .steps()
            .iter()
            .all(|step| step.events() == &UPDATE_EVENT_ORDER),
        "operation order changed",
    )?;
    require(
        first.result.steps().iter().any(|step| step.clipped()),
        "fixture no longer exercises clipping",
    )?;
    require(
        first
            .result
            .checkpoints()
            .last()
            .unwrap()
            .train()
            .mean_loss()
            < first.result.checkpoints()[0].train().mean_loss(),
        "training loss did not improve",
    )?;
    require(
        first.result.checkpoints().iter().all(|checkpoint| {
            checkpoint.train().recorded_graphs() == 0
                && checkpoint.validation().recorded_graphs() == 0
        }),
        "evaluation recorded a graph",
    )?;
    require(
        first.result.final_optimizer().step_count() == LEARNING_RATES.len() as u64,
        "optimizer did not execute every scheduled step",
    )?;
    let selected = first
        .result
        .checkpoints()
        .iter()
        .filter(|checkpoint| checkpoint.selected())
        .collect::<Vec<_>>();
    require(selected.len() == 1, "selection marker count changed")?;
    require(
        first.test_partition_rejected,
        "test partition was not rejected during preflight",
    )?;
    require(
        selected[0].step() == first.result.selected_step(),
        "selected checkpoint and restored state disagree",
    )?;
    require(
        first.result.selected_state().bit_pattern()
            == parameter_bits(first.result.selected_model()),
        "restored selected model changed snapshot bits",
    )?;
    let replay_bitwise = replay_equal(&first, &second);
    let test_partition_rejected = first.test_partition_rejected;
    Ok(LearnerEvidence {
        result: first.result,
        replay_bitwise,
        input_model_unchanged: first.input_model_unchanged,
        input_optimizer_unchanged: first.input_optimizer_unchanged,
        test_partition_rejected,
        test_reads: usize::from(!test_partition_rejected),
        history: historical_selection(),
    })
}

Print only the frozen Chapter 33 learner report rust/demos/ch33-training-selection/src/main.rs

fn main() {
    print!(
        "{}",
        ch33_training_selection::learner_report().expect("Chapter 33 fixture must remain valid")
    );
}

Run cargo run --quiet --locked -p ch33-training-selection. The report lists all five measured checkpoints, the selected validation state, clipping, preserved-node and explicit gradient-clear evidence, the rejected test-partition probe, ownership checks, and bitwise replay.

Read measured checkpoints without inventing a curve

The recorded run contains the fixed configuration, the six-operation order repeated for each of eight updates, four schedule segments, all eight update records, an axis derived from the measured losses, five checkpoint pairs, one selection record, and the verified boundaries. Together they show the relationship between optimization and model selection without filling in unmeasured steps.

Record the measured updates, checkpoints, selection, and proof fields rust/demos/ch33-training-selection/src/diagram_trace.rs#training-selection-trace

fn axis(losses: impl Iterator<Item = f64>) -> (f64, f64, [f64; 3]) {
    let values = losses.collect::<Vec<_>>();
    let smallest = values.iter().copied().fold(f64::INFINITY, f64::min);
    let largest = values.iter().copied().fold(f64::NEG_INFINITY, f64::max);
    let minimum = (smallest * 10.0).floor() / 10.0;
    let mut maximum = (largest * 10.0).ceil() / 10.0;
    if maximum <= minimum {
        maximum = minimum + 0.1;
    }
    let midpoint = (minimum + maximum) / 2.0;
    (minimum, maximum, [minimum, midpoint, maximum])
}

pub fn diagram_trace() -> Result<String, FixtureError> {
    let evidence = learner_evidence()?;
    let result = &evidence.result;
    let mut lines = vec![
        "TRAINING_SELECTION_TRACE_V1".to_owned(),
        format!(
            "CONFIG|seed=33|updates={}|batch_size=2|context=2|validation_every=2|clip_norm={MAX_GRADIENT_NORM:.6}|runtime_limit_ms={RUNTIME_LIMIT_MS}",
            LEARNING_RATES.len()
        ),
        "ORDER|events=forward>backward>finite-check>clip>adamw-step>zero-grad".to_owned(),
        format!(
            "SCHEDULE|start=1|end=2|learning_rate={:.6}",
            LEARNING_RATES[0]
        ),
        format!(
            "SCHEDULE|start=3|end=4|learning_rate={:.6}",
            LEARNING_RATES[2]
        ),
        format!(
            "SCHEDULE|start=5|end=6|learning_rate={:.6}",
            LEARNING_RATES[4]
        ),
        format!(
            "SCHEDULE|start=7|end=8|learning_rate={:.6}",
            LEARNING_RATES[6]
        ),
    ];
    for step in result.steps() {
        lines.push(format!(
            "UPDATE|step={}|batch={}|learning_rate={:.6}|train_loss={:.6}|grad_norm_before={:.6}|grad_norm_after={:.6}|clipped={}|finite={}|nodes_preserved={}|cleared={}",
            step.step(),
            step.batch_windows().join(","),
            step.learning_rate(),
            step.train_loss(),
            step.gradient_norm_before(),
            step.gradient_norm_after(),
            step.clipped(),
            step.finite_gradients(),
            step.parameter_nodes_preserved(),
            step.cleared_gradients()
        ));
    }
    let (minimum, maximum, ticks) = axis(result.checkpoints().iter().flat_map(|checkpoint| {
        [
            checkpoint.train().mean_loss(),
            checkpoint.validation().mean_loss(),
        ]
    }));
    lines.push(format!(
        "AXIS|min={minimum:.6}|max={maximum:.6}|ticks=[{:.6},{:.6},{:.6}]",
        ticks[0], ticks[1], ticks[2]
    ));
    for checkpoint in result.checkpoints() {
        lines.push(format!(
            "CHECKPOINT|step={}|train_loss={:.6}|validation_loss={:.6}|selected={}|train_graphs={}|validation_graphs={}",
            checkpoint.step(),
            checkpoint.train().mean_loss(),
            checkpoint.validation().mean_loss(),
            checkpoint.selected(),
            checkpoint.train().recorded_graphs(),
            checkpoint.validation().recorded_graphs()
        ));
    }
    lines.extend([
        format!(
            "SELECT|step={}|validation_loss={:.6}|criterion=validation-only|snapshot=true|test_partition_rejected={}",
            result.selected_step(),
            result.selected_validation_loss(),
            evidence.test_partition_rejected
        ),
        format!(
            "PROOF|fixed_seed_batches=true|schedule_exact=true|finite_gradients=true|parameter_nodes_preserved=true|cleared_gradients=true|clipping_observed={}|train_loss_decreased=true|validation_no_grad=true|selection_matches_argmin=true|test_partition_rejected={}|replay_bitwise={}|input_unchanged={}",
            result.steps().iter().any(|step| step.clipped()),
            evidence.test_partition_rejected,
            evidence.replay_bitwise,
            evidence.input_model_unchanged && evidence.input_optimizer_unchanged
        ),
        "END_TRAINING_SELECTION_TRACE".to_owned(),
    ]);
    Ok(lines.join("\n") + "\n")
}

Separate updates from validation-based selection

Repeat the same six-operation order for each of eight updates, then compare ten isolated train and validation measurements at five checkpoints without inventing a curve between them.

Circle: measured training loss
Diamond: measured validation loss
Double underline: selected validation state
Gaps contain no measured values or interpolated line

One successful update has six ordered operations

The trainer releases the backward graph, checks every gradient, clips once across the complete decoder, commits the predetermined-rate update into the existing parameter nodes, and clears their raw gradients explicitly.

Training forward
Backward and release
Finite-gradient check
Global-norm clip
Scheduled AdamW step
Explicitly clear raw gradients

Measure candidates only at declared checkpoints

The table and markers preserve the exact six-decimal values. Only observations at steps 0, 2, 4, 6, and 8 exist; their visual spacing is presentation, not new arithmetic.

Measure candidates only at declared checkpoints

Scrollable training and validation checkpoint evidence
Step	Training loss	Validation loss	Selected state
$s=0$	$2.095016$	$1.918167$	Not selected
$s=2$	$1.562026$	$1.696310$	Not selected
$s=4$	$1.453259$	$1.687788$	Not selected
$s=6$	$1.369832$	$1.642599$	Not selected
$s=8$	$1.322897$	$1.595297$	Selected

Keep optimization, selection, and test evidence separate

Validation can choose a snapshot but cannot update it; this trainer rejects Test throughout the training execution, before the next chapter creates its local evaluator.

Earliest validation minimum

$s^*=8$

$\mathcal{L}_{va}=1.595297$

criterion=validation-only, test_partition_rejected=true

Train and validation record no graphs

$0\to0$ Verified by the recorded run

train_graphs=0, validation_graphs=0

Partition information boundary

Train → Training forward

Validation → Earliest validation minimum

Test → Rejected before any update

Clipping, schedule, and replay

$\lVert \widetilde g_s\rVert_2\leq0.35$ Clipped before the optimizer step

$\eta_s\in\{0.040,0.025,0.015,0.008\}$

Original parameter nodes preserved parameter_nodes_preserved=true

Raw gradients explicitly cleared cleared_gradients=true

schedule_exact=true, replay_bitwise=true

There are ten isolated markers: one training and one validation observation at each of five checkpoints. No segment joins them. A line would imply values at unmeasured steps, so the exact table remains the authoritative comparison.

At $s=0$ , the losses are $2.095016$ and $1.918167$ . At $s=8$ , they are $1.322897$ and $1.595297$ . The double-underlined validation marker identifies $s^*=8$ because $1.595297$ is the smallest measured validation loss. The selection record also says criterion=validation-only, snapshot=true, and test_partition_rejected=true.

Every update in this particular fixture has raw gradient norm above $0.35$ and therefore reaches AdamW with norm $0.35$ . That observation is fixture evidence, not a requirement that all real LLM updates clip. The generic trainer also accepts finite gradients already below the ceiling.

Each update record also says nodes_preserved=true and cleared=true. The summary proof repeats the whole-run invariant as parameter_nodes_preserved=true and cleared_gradients=true. The first field means AdamW committed through the original parameter nodes; the second means the trainer explicitly cleared their raw gradients afterward.

Test the information boundary and state ownership

Change the update epoch from Train to Test. Predict which preflight error occurs and why neither the caller’s decoder nor optimizer may change.
Replace checkpoint steps $[0,2,4,6,8]$ with $[0,2,4,6]$ . Explain why a final candidate is required after all planned updates.
Construct validation losses $[1.8,1.6,1.6]$ . Predict which checkpoint strict comparison retains and relate it to the definition of $s^*$ .
Average two batch means without token weights when their token counts differ. Compare that result with the token-weighted formula above.
Let AdamW commit new values into the existing parameter nodes. Explain why the registry, decoder components, and tied output projection all observe the update without a decoder rebuild.
Draw a line between the five validation markers. List the unmeasured claims that line introduces, then remove it and recover the discrete evidence.

Check your reasoning

Test is rejected before a forward pass because it cannot update or select.
The checkpoint schedule must include the final planned update so every completed run has a final candidate.
The first loss equal to the minimum wins because replacement uses strict less-than comparison.
Batch means need weights proportional to their predicted-token counts.
Those handles alias the same TensorValue nodes. Updating each node’s value preserves the aliases, including the shared embedding/output node, so the next forward pass reads the committed values.
Only the ten measured markers and exact table are justified; values between checkpoints remain unknown.

Freeze the selected decoder before local test evaluation

The cumulative decoder can now execute a complete bounded training plan and return a frozen validation-selected state. Chapter 34 will give one local evaluator instance post-selection access to the test fixture and compare that state fairly with the frozen baseline.

That next step must not tune the schedule, clipping ceiling, number of updates, or checkpoint choice after seeing test results. Chapter 33 ends with model selection complete and a verified rejection of Test at the training boundary. Chapter 34’s local evaluator will use its one access for evaluation rather than selection. That count applies to one evaluator instance in one execution, not to the repository’s complete history of checking the fixture.

Plan every update before you train

Update on train, choose on validation

Keep steps, gradients, and partitions distinct

From training-only reports to validation-selected LLM checkpoints

Represent the complete training plan explicitly in Rust

Read measured checkpoints without inventing a curve

Separate updates from validation-based selection

One successful update has six ordered operations

Training forward

Backward and release

Finite-gradient check

Global-norm clip

Scheduled AdamW step

Explicitly clear raw gradients

Measure candidates only at declared checkpoints

Measure candidates only at declared checkpoints

Keep optimization, selection, and test evidence separate

Earliest validation minimum

Train and validation record no graphs

Partition information boundary

Clipping, schedule, and replay

Test the information boundary and state ownership

Freeze the selected decoder before local test evaluation