Schedule AdamW updates and clip global gradients

Control gradient magnitude and the rate of each update

A backward pass can produce a gradient dominated by a few large coordinates. Feeding that gradient directly into the optimizer lets its magnitude determine the optimizer’s new moment values. Independently, a fixed learning rate gives the same multiplier to the first and last updates. A training experiment may instead call for a gradual increase at the start and smaller rates near the end. These are two separate choices: control the gradient entering AdamW, and choose the rate used by each successful update.

The existing trainer already performs global gradient-norm clipping and AdamW. We will inspect that clipping operation and extend its existing learning-rate schedule with linear warmup followed by one cosine decay. The optimizer, model and update owner stay the same.

One factor for gradients in two tensors

Start with a constructed scalar diagnostic on the text model. Its derivative has values $3$ and $4$ in the first two coordinates of token_embedding.weight, and $12$ in the first coordinate of blocks.0.attention_norm.gain. Every other parameter-gradient coordinate is zero. This is an inspection objective chosen to make its gradient easy to check; it is not a next-token likelihood.

The complete gradient has Euclidean norm $\sqrt{3^2+4^2+12^2}=\sqrt{169}=13$. Set the global norm limit to $6.5$. Multiplying every coordinate by the same factor $6.5/13=0.5$ gives $1.5$, $2$ and $6$, whose combined norm is $6.5$. The nonzero vector keeps its direction. Clipping each tensor independently would change the relationship between the two tensors’ contributions.

The Rust diagnostic observes exactly those three effective gradient values, applies one existing AdamW update at rate $0.001$, and clears gradients. It uses its own fresh model and optimizer; the text-training experiment below starts separately.

Six successful updates, with two warmup updates

For the text experiment, plan six successful optimizer updates. Choose a peak rate of $0.002$, a final decay rate of $0.0002$, and two warmup updates. Linear warmup uses half the peak, $0.001$, on update $1$ and reaches $0.002$ on update $2$. The four following updates move along one cosine decay to the final rate.

The recorded rates for updates $3$, $4$ and $5$ are $0.0017363961030678928$, $0.0011$ and $0.0004636038969321073$. Update $6$ uses $0.0002$. These are the reported f64 values, rather than a claim that decimal arithmetic or every backend gives identical bits. A successful update owns one rate; an attempted call that is refused or skipped does not consume a rate.

Define one global clip and one warmup–cosine schedule

Let $j$ enumerate scalar coordinates across all distinct named parameter tensors, counting a tied parameter once. Let $g_j$ be a gradient coordinate after any loss unscaling and before clipping. The complete gradient’s Euclidean norm is $r$, its finite positive limit is $c$, and $a$ is the one common nonamplifying factor. AdamW receives effective coordinate $q_j$.

For the schedule, $t$ is the one-based successful optimizer-update index and $T$ is the total planned number of those updates. $W$ counts warmup updates; $P$ is the positive peak rate; $F$ is the positive final decay rate; and $\eta_t$ is the rate assigned to update $t$. The constant $\pi$ is the usual circle constant, and the cosine angles use radians. The ideal rules are

The zero-norm branch avoids division by zero and leaves the gradient zero. For a nonzero gradient, the factor is $1$ when the norm already fits and $c/r$ when it exceeds the limit. Thus in exact arithmetic the effective gradient has norm at most $c$, and a nonzero vector retains its direction.

With positive warmup, $t/W$ reaches $1$ on update $W$. The subsequent cosine argument moves from $0$ toward $\pi$ as updates approach $T$, moving the rate from $P$ to $F$. With zero warmup, update $1$ must already use $P$; the cosine progress therefore starts from $t-1$ over $T-1$. There is one decay and no restart. $F$ is the decay endpoint, not a lower bound on rates earlier in warmup.

The warmup–cosine constructor requires $2\le T\le2^{53}$, $0\le W<T$ and finite rates $0<F\le P$. The integer bound lets every index convert exactly to f64; leaving $W<T$ reserves a decay update and a nonzero denominator. Allocation must also succeed. The Rust code assigns peak and final endpoints explicitly and passes every generated rate through the existing positive-rate validator. Even positive ideal warmup values can underflow to stored zero and be refused.

The existing norm calculation rescales its sum of squares to avoid avoidable overflow. Its gradient transform supports both uniform multiplication and a normalized representation for numerical range. The equations describe the ideal transformation; the implementation still performs represented f64 operations. When clipping occurs, the trainer’s after-norm field records the chosen limit. That field is not an independent remeasurement of every effective coordinate. The three-coordinate diagnostic above supplies a separate hand calculation of its resulting norm.

Keep clipping, Adam moments and weight decay separate

Clipping limits the gradient supplied to AdamW’s moments. It does not directly limit the final parameter displacement, because AdamW also uses coordinatewise moment normalization and a separate weight-decay term.

For one coordinate, write its effective gradient at update $t$ as $q_t$. Let $m_t$ and $v_t$ be its first and second moments, with initial $m_0=v_0=0$. Their coefficients $\beta_1$ and $\beta_2$ lie in $[0,1)$; the bias-corrected moments are $\widehat m_t$ and $\widehat v_t$. Let $\epsilon>0$ be the denominator offset, $\theta_{t-1}$ the old weight, and $\lambda$ its decay coefficient. For a coordinate selected for decay, the existing AdamW rule is, in exact arithmetic,

Schedule decay lowers the learning rate $\eta_t$ across updates. Weight decay subtracts $\eta_t\lambda\theta_{t-1}$ during an update, outside the loss-gradient and moment calculation. Do not add that decay term to the gradient before clipping. Both subtractions use the scheduled rate, but they perform different operations.

The diagnostic and six-update text run use $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-8}$ and $\lambda=0.01$. In the diagnostic’s first coordinate, the effective gradient is $1.5$. The stored first and second moments are $0.14999999999999997$ and $0.1125000000000001$; the corrected moments are $1.5$ and $2.25$. At rate $0.001$, the adaptive subtraction is $0.0009999999933333335$ and the separate decay subtraction is $9.270782062966737\times10^{-7}$. Starting from weight $0.09270782062966737$, the stored result is $0.09170689355812774$. These represented values expose the two branches; the cap on the input gradient is not a cap on their combined parameter change.

From recurrent-gradient control to scheduled transformer training

Exploding gradients made recurrent-network optimization difficult; controlling their norm addresses gradient size rather than defining a complete training schedule. Pascanu, Mikolov and Bengio propose gradient norm clipping for exploding gradients in recurrent neural networks. Their paper, On the difficulty of training Recurrent Neural Networks, was first submitted in 2012; the cited revision is from 2013. This supports the norm-control idea, not a guarantee about vanishing gradients or AdamW convergence.

Loshchilov and Hutter separate weight decay from the loss-gradient update in adaptive optimization. Decoupled Weight Decay Regularization was first submitted in 2017; the cited revision is from 2019. That distinction is why this lesson leaves decay outside the clipping and moment branch and reuses the course’s existing AdamW implementation.

LLaMA combines AdamW with gradient clipping and a learning-rate schedule for transformer language-model training. Touvron and colleagues’ 2023 LLaMA report uses AdamW, a cosine schedule ending at one tenth of the maximum rate, two thousand warmup steps and gradient clipping with threshold one. See its optimizer section. Our six-update experiment and two warmup updates are a small teaching choice. Its text-run norm limit $0.5$ and decay coefficient $0.01$ also belong to this experiment; this run does not reproduce LLaMA training.

In this decoder, clipping controls the gradient entering AdamW, while the schedule chooses the rate associated with each successful optimizer update. Reuse the existing global clipping and AdamW update, adding linear warmup followed by one cosine decay over successful optimizer updates. The Rust example makes norm control, scheduled rates and preserved state inspectable on tiny text-model fixtures; it does not reproduce the papers’ training runs or establish comparative language-model quality.

Extend rate construction and reuse the update owner

LearningRateSchedule::linear_warmup_cosine_decay returns the same schedule type as the explicit-rate constructor. Its vector contains one rate per planned successful update, with no mutable cursor. Changing a rate does not recreate AdamW or reset its moments.

impl LearningRateSchedule {
    /// Builds one linear warmup and one cosine decay over successful updates.
    ///
    /// Updates are numbered `1..=total_steps`. Positive warmup reaches the peak
    /// at `warmup_steps`; zero warmup starts at the peak on update one. The last
    /// update always uses `final_rate`, which is a decay endpoint rather than a
    /// lower bound on the warmup rates. Both endpoints must be positive under
    /// the existing AdamW policy. There are no restarts or mutable schedule
    /// counters: callers select the next successful optimizer-update index.
    pub fn linear_warmup_cosine_decay(
        total_steps: usize,
        warmup_steps: usize,
        peak_rate: f64,
        final_rate: f64,
    ) -> Result<Self, TrainerError> {
        let invalid = |reason| TrainerError::InvalidWarmupCosineSchedule { reason };
        if total_steps < 2 || total_steps as u128 > (1_u128 << 53) {
            return Err(invalid("total steps must be between 2 and 2^53"));
        }
        if warmup_steps >= total_steps {
            return Err(invalid("warmup must leave at least one decay update"));
        }
        if !peak_rate.is_finite() || peak_rate <= 0.0 {
            return Err(invalid("the peak rate must be finite and positive"));
        }
        if !final_rate.is_finite() || final_rate <= 0.0 || final_rate > peak_rate {
            return Err(invalid(
                "the final rate must be finite, positive and no greater than the peak",
            ));
        }
        let mut rates = Vec::new();
        rates
            .try_reserve_exact(total_steps)
            .map_err(|_| invalid("the schedule cannot be allocated"))?;
        for step in 1..=total_steps {
            let rate = if step == total_steps {
                final_rate
            } else if step == warmup_steps || (warmup_steps == 0 && step == 1) {
                peak_rate
            } else if step < warmup_steps {
                peak_rate * (step as f64 / warmup_steps as f64)
            } else {
                let (elapsed, duration) = if warmup_steps == 0 {
                    (step - 1, total_steps - 1)
                } else {
                    (step - warmup_steps, total_steps - warmup_steps)
                };
                let progress = elapsed as f64 / duration as f64;
                let weight = 0.5 * (1.0 + (std::f64::consts::PI * progress).cos());
                final_rate + (peak_rate - final_rate) * weight
            };
            rates.push(rate);
        }
        // Preserve the existing positive-rate validation, including a warmup
        // value that rounds to zero even though its mathematical value is positive.
        Self::new(rates)
    }
}

The diagnostic constructs its scalar objective from existing parameter leaves. Coefficients make the two selected tensors contribute the explained gradient coordinates; the existing reverse engine calculates their derivatives.

/// A scalar diagnostic on two existing model tensors gives gradients 3,4 and12.
/// This is an inspection objective, separate from the text likelihood below.
fn constructed_gradient_loss(
    model: &DecoderModel,
    context: AutogradContext<'_>,
) -> Result<TensorValue, TrainerError> {
    let mut terms = Vec::new();
    for (parameter, nonzero) in model
        .parameters()
        .iter()
        .zip([&[3.0, 4.0][..], &[12.0][..]])
    {
        let count = parameter.tensor().value().as_slice().len();
        let mut coefficients = vec![0.0; count];
        coefficients[..nonzero.len()].copy_from_slice(nonzero);
        let coefficients = TensorValue::constant(Tensor::from_vec(vec![count], coefficients)?)?;
        terms.push(
            parameter
                .tensor()
                .reshape_with_context(context, &[count])?
                .mul_with_context(context, &coefficients)?
                .sum_axis_with_context(context, 0, false)?,
        );
    }
    Ok(terms[0].add_with_context(context, &terms[1])?)
}

The probe passes that scalar through decoder_update_with_forward. It observes the existing trainer’s common gradient transform and AdamW’s effective gradient, moment and decay trace, then checks the cleared gradients. The transform supplies effective moment inputs without overwriting the raw gradient buffers; the trainer clears those buffers after the update commits. In scaled paths, unscaling and clipping can be combined in that transform rather than performed as separate in-place changes.

/// Observe the existing trainer's common transform across two named tensors.
pub fn clipping_probe() -> DemoResult<Value> {
    let model = model_fixture()?;
    let mut optimizer = optimizer()?;
    let plan = TrainerConfig::new(LearningRateSchedule::new(vec![0.001])?, vec![0, 1], 6.5)?;
    let execution = TensorExecution::cpu();
    let mut observer = Observe::default();
    let update = decoder_update_with_forward(
        &model,
        &mut optimizer,
        &plan,
        1,
        || {
            constructed_gradient_loss(
                &model,
                AutogradContext::recording().with_execution(&execution),
            )
        },
        &mut observer,
    )?;
    let trace = update.optimizer_trace.ok_or("missing clipping trace")?;
    let first = &trace.updates()[0];
    let second = &trace.updates()[1];
    let second_offset = model.parameters()[0].tensor().value().as_slice().len();
    let raw = [0, 1, second_offset].map(|index| f64::from_bits(observer.raw_gradient_bits[index]));
    let effective = [
        first.gradient()[0],
        first.gradient()[1],
        second.gradient()[0],
    ];
    if raw != [3.0, 4.0, 12.0] || effective != [1.5, 2.0, 6.0] {
        return Err("constructed gradients or shared clipping changed".into());
    }
    Ok(json!({
        "objective":"constructed scalar diagnostic, not text likelihood",
        "parameter_names":[first.name(),second.name()],
        "raw_gradients":raw,
        "global_norm_before":update.step.gradient_norm_before(),
        "clip_threshold":6.5,
        "global_norm_after":update.step.gradient_norm_after(),
        "clipped":update.step.clipped(),
        "transform":transform_record(observer.transform.ok_or("missing transform")?),
        "effective_gradients":effective,
        "learning_rate":trace.learning_rate(),
        "optimizer_step":trace.step(),
        "gradients_cleared":gradients(&model).iter().all(|&bits|bits==0),
        "first_coordinate":{
            "first_moment":first.first_moment()[0],
            "second_moment":first.second_moment()[0],
            "corrected_first_moment":first.corrected_first_moment()[0],
            "corrected_second_moment":first.corrected_second_moment()[0],
            "weight_before":first.before()[0],
            "adaptive_delta":first.adaptive_delta()[0],
            "decay_delta":first.decay_delta()[0],
            "weight_after":first.after()[0],
            "weight_decay":first.effective_weight_decay()
        }
    }))
}

Advance the clock only when AdamW commits

The requested update must equal the optimizer’s committed count plus one. This check runs before the forward callback. Looking up a rate does not consume it.

fn validate_update_step(
    optimizer: &AdamW,
    config: &TrainerConfig,
    step: usize,
) -> Result<(f64, u64), TrainerError> {
    let learning_rate = config
        .schedule
        .learning_rate(step)
        .ok_or(TrainerError::InvalidUpdateStep { step })?;
    let expected = u64::try_from(step).map_err(|_| TrainerError::InvalidUpdateStep { step })?;
    if optimizer.step_count().checked_add(1) != Some(expected) {
        return Err(TrainerError::OptimizerStepMismatch {
            expected,
            actual: optimizer.step_count(),
        });
    }
    Ok((learning_rate, expected))
}

The text run reuses Chapter 11’s seed-$39$ model fixture: vocabulary $258$, residual width $4$, one attention head, feed-forward width $8$, two decoder blocks and capacity for $8$ positions. It has $20$ named parameter tensors and $1372$ learned scalars. Input IDs [0,99,100] have shape $[1,3]$, meaning one row and three positions; targets [99,100,1] provide three next-token targets for the mean negative log likelihood. The same fixed text is reused at every update.

Within this run, one live model and one AdamW instance continue through all six generated rates. Computation is CPU f64, dropout is disabled and the loss scale is $1$. The unscaled entry and the scaled/accumulated entries share the same finish owner. After contributions combine, its logical operations are unscaling when applicable, checking the finite global gradient, clipping once, applying scheduled AdamW once and clearing gradients. The gradient transform can fuse unscaling and clipping before the moment calculation. Microbatches and recomputed forwards do not create extra schedule steps.

After committing update $1$, the run deliberately requests update $3$. It receives OptimizerStepMismatch with "optimizer committed step 1, expected 3", and forward_called is false. The error’s expected index comes from the requested $3$; the next eligible update is still $1+1=2$. Weight and gradient bits, parameter revisions, populated moments, beta powers and optimizer step remain unchanged. The default schedule still supplies $0.002$ for update $2$. The run then actually executes update $2$ and continues through $6$. Executing the next valid update is not retrying the refused request for $3$.

/// Reuse one live model and optimizer across all six generated learning rates.
pub fn text_training_probe(warmup_steps: usize) -> DemoResult<Value> {
    let rates = schedule(warmup_steps)?;
    let plan = TrainerConfig::new(rates.clone(), vec![0, TOTAL_STEPS], 0.5)?;
    let model = model_fixture()?;
    let mut optimizer = optimizer()?;
    let execution = TensorExecution::cpu();
    let mut updates = Vec::new();
    let mut refusal = None;
    for step in 1..=TOTAL_STEPS {
        if step == 2 {
            let before = DecoderModelState::snapshot(&model).bit_pattern();
            let before_gradients = gradients(&model);
            let before_revisions = revisions(&model);
            let before_optimizer = optimizer.persistence_state();
            let next_rate_before = rates.learning_rate(2);
            let mut forward_called = false;
            let rejected = decoder_update_with_forward(
                &model,
                &mut optimizer,
                &plan,
                3,
                || {
                    forward_called = true;
                    Err(TrainerError::ObserverRejected(
                        "out-of-order forward was called",
                    ))
                },
                &mut NoDecoderUpdateObserver,
            );
            let error = match rejected {
                Err(
                    error @ TrainerError::OptimizerStepMismatch {
                        expected: 3,
                        actual: 1,
                    },
                ) => error,
                _ => return Err("out-of-order update was not refused before forward".into()),
            };
            let preserved = before == DecoderModelState::snapshot(&model).bit_pattern()
                && before_gradients == gradients(&model)
                && before_revisions == revisions(&model)
                && same_optimizer_bits(&before_optimizer, &optimizer.persistence_state());
            if forward_called || !preserved || next_rate_before != rates.learning_rate(2) {
                return Err("refused update changed live training state or schedule lookup".into());
            }
            refusal = Some(json!({
                "attempted_step":3,"optimizer_step_before":1,"optimizer_step_after":optimizer.step_count(),
                "error":error.to_string(),"forward_called":forward_called,
                "weights_gradients_revisions_moments_powers_step_preserved":preserved,
                "next_valid_step":2,"next_valid_learning_rate":next_rate_before
            }));
        }
        let previous = optimizer.persistence_state();
        let first_name = model.parameters()[0].name();
        let previous_first_moment = previous
            .state(first_name)
            .map_or(0.0, |m| m.first_moment()[0]);
        let mut observer = Observe::default();
        let update = decoder_update_with_forward(
            &model,
            &mut optimizer,
            &plan,
            step,
            || {
                Ok(model.loss_with_context(
                    AutogradContext::recording().with_execution(&execution),
                    &[0, 99, 100],
                    &[1, 3],
                    &[99, 100, 1],
                )?)
            },
            &mut observer,
        )?;
        let trace = update.optimizer_trace.ok_or("missing text-update trace")?;
        let first = &trace.updates()[0];
        updates.push(json!({
            "step":trace.step(),"learning_rate":trace.learning_rate(),
            "train_loss":update.step.train_loss(),
            "gradient_norm_before":update.step.gradient_norm_before(),
            "gradient_norm_after":update.step.gradient_norm_after(),
            "clipped":update.step.clipped(),
            "gradients_cleared":gradients(&model).iter().all(|&bits|bits==0),
            "first_coordinate":{
                "parameter":first.name(),"raw_gradient":f64::from_bits(observer.raw_gradient_bits[0]),
                "effective_gradient":first.gradient()[0],"previous_first_moment":previous_first_moment,
                "first_moment":first.first_moment()[0],"second_moment":first.second_moment()[0],
                "weight_before":first.before()[0],"adaptive_delta":first.adaptive_delta()[0],
                "decay_delta":first.decay_delta()[0],"weight_after":first.after()[0]
            }
        }));
    }
    Ok(json!({
        "input_ids":[0,99,100],"target_ids":[99,100,1],"input_shape":[1,3],
        "valid_targets_per_update":3,"updates":updates,
        "refused_update":refusal,"final_optimizer_step":optimizer.step_count(),
        "optimizer_base_rate_unchanged":optimizer.config().learning_rate()==PEAK_RATE,
        "beta1":optimizer.config().beta1(),"beta2":optimizer.config().beta2(),
        "epsilon":optimizer.config().epsilon(),"weight_decay":optimizer.config().weight_decay(),
        "clip_threshold":0.5,"named_parameter_tensors":model.parameters().len(),
        "scalar_parameters":model.parameter_count(),"dropout":"disabled","loss_scale":1.0
    }))
}

Every default text update is marked clipped at threshold $0.5$ and clears its gradients. The first measured before-norm is $1.2630253079143638$; the trainer reports after-norm $0.5$ as the chosen clipping cap. Its first recorded forward mean loss is $5.668398842431571$, before update $1$. The sixth recorded forward mean loss is $5.610984283272283$, before update $6$. There is no loss measurement after the sixth commit and no held-out or comparative model-quality result.

The first coordinate of token_embedding.weight makes moment continuity visible. Update $1$ receives effective gradient $0.07471210553828579$ and stores first moment $0.0074712105538285774$. Update $2$ reads that same previous moment, receives effective gradient $0.062260867053383434$ and stores $0.012950176203784062$, using rate $0.002$. AdamW reaches step $6$, while its base configuration rate remains $0.002$; the per-update override changes the rate used for that step without resetting optimizer state.

Inspect actual same-step retries in a separate fixed-rate fixture

The demonstration also calls Chapter 12’s existing public overflow_fixture. That experiment creates another model and optimizer, retains the fixed rates [0.001,0.001], and uses the existing CPU FP16/F32 matrix policy and its original settings, including $\beta_2=0.999$. These rates are separate from the generated cosine schedule.

After successful update $1$ populates moments, scheduled update $2$ overflows and is skipped eight times. The first skip lowers the loss scale from $16777216$ to $8388608$ and preserves weight and gradient bits, parameter revisions, AdamW moments, beta powers and optimizer step $1$. The fixture actually retries that same scheduled update $2$; it succeeds at scale $65536$, commits optimizer step $2$ and clears gradients.

A numerical skip performs no AdamW update or decoupled weight decay and consumes no learning-rate step. Its loss-scale run state can change. This state boundary does not promise rollback of arbitrary caller effects, a data cursor or live random-number state. Applying AdamW to zero gradients would still be an optimizer step and could decay weights; an overflow skip does neither.

pub fn demonstrate(warmup_steps: usize) -> DemoResult<Value> {
    let rates = schedule(warmup_steps)?;
    Ok(json!({
        "chapter":"14-optimizer-schedules-clipping","execution":"cpu-f64-reference",
        "schedule":{"steps":TOTAL_STEPS,"warmup_steps":warmup_steps,
            "peak_rate":PEAK_RATE,"final_rate":FINAL_RATE,"rates":rates.rates(),
            "clock":"successful optimizer updates, numbered from one"},
        "clipping":clipping_probe()?,"text_training":text_training_probe(warmup_steps)?,
        "reused_fixed_rate_overflow":{
            "source":"Chapter12 overflow_fixture; separate CPU FP16/F32 experiment",
            "scheduled_rates":[0.001,0.001],"observation":overflow_fixture()?
        }
    }))
}

Follow rates, one global factor and the next valid update

Reproduce the rates and inspect the update boundaries

The existing clap CLI accepts --warmup-steps from $0$ through $5$ and defaults to $2$. Each command below starts its own demonstration and prints one JSON record; separate invocations do not continue each other’s optimizer.

use clap::Parser;

#[derive(Parser)]
#[command(about = "Inspect six scheduled text updates and global gradient clipping")]
struct Args {
    /// Number of warmup updates in the six-update example.
    #[arg(long, default_value_t = 2, value_parser = clap::value_parser!(u8).range(0..=5))]
    warmup_steps: u8,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args = Args::parse();
    println!(
        "{}",
        practical_ch14_optimizer_schedules_clipping::demonstrate(usize::from(args.warmup_steps))?
    );
    Ok(())
}

From the repository root, with the existing course Docker setup available, run the default CPU example. It needs no GPU, external corpus or downloaded model:

COURSE_CORPUS=false ./course run cargo run --quiet --offline --locked -p practical-ch14-optimizer-schedules-clipping

The record has schedule, clipping, text_training and reused_fixed_rate_overflow objects. The default schedule.rates starts with [0.001,0.002] and ends with 0.0002. text_training.updates contains six committed steps, each with gradients_cleared: true. text_training.refused_update.forward_called is false. The reused overflow observation reports eight skips and a successful scheduled-step-two retry.

Keep the same repository-root working directory and existing Docker setup to run both boundary warmup choices and the focused tests:

COURSE_CORPUS=false ./course run cargo run --quiet --offline --locked -p practical-ch14-optimizer-schedules-clipping -- --warmup-steps 0
COURSE_CORPUS=false ./course run cargo run --quiet --offline --locked -p practical-ch14-optimizer-schedules-clipping -- --warmup-steps 5
COURSE_CORPUS=false ./course run cargo test --offline --locked -p practical-ch14-optimizer-schedules-clipping
COURSE_CORPUS=false ./course run cargo test --offline --locked -p llm-from-scratch-practical warmup_cosine

Both variants print six successful text updates and finish at rate $0.0002$. The zero-warmup variant peaks on update $1$; the five-update warmup variant peaks on update $5$. The demo test command runs three library tests for the shared clip, continuing optimizer and supported warmup boundaries. The warmup_cosine selector runs three core constructor tests, covering rates, endpoint conventions, invalid inputs and warmup underflow refusal.

The following practice is optional; the explanations above supply the rules and observed results.

1. Explain the diagnostic’s shared clipping factor for gradient coordinates $3$, $4$ and $12$ across two tensors, using global limit $6.5$. Identify the effective coordinates and explain why this scalar diagnostic is separate from text likelihood.
2. Use the zero- and five-warmup commands and setup above. Inspect schedule.rates and text_training.final_optimizer_step; identify each peak update and the common final update and rate.
3. Use the default command and setup above. Inspect the first two text_training.updates records for first-moment continuity, then inspect clipping.first_coordinate for the adaptive and weight-decay subtractions. Explain which quantity global clipping bounds.
4. Use the default command and setup above. Compare text_training.refused_update with reused_fixed_rate_overflow.observation. Identify the next valid update after refusal and the actual same-step retry after numerical skips, retaining each experiment’s rates.

Preserve the update clock before preserving model state

The text encoder, source-independent transformer, tied head and existing AdamW now share generated rates indexed by committed logical updates. Global clipping still transforms the complete gradient before moments; refused and overflow-skipped attempts consume no optimizer step. The backbone continues to consume vectors, positions and attention layout without requiring a tokenizer or vocabulary. Text remains the worked input and output.

Chapter 15 will address portable model saving and loading in a fresh process, followed by the additional state needed for training continuation. Saving the learned composition, weights, settings and required assets for inference is a different responsibility from retaining optimizer, random-generator and data position state for exact continuation.

Full-corpus training, physical-GPU measurements and accelerated mixed-precision execution still need their own evidence. The reused CPU FP16/F32 fixture is a precision experiment, not a physical-GPU result.