Save, load and resume the composed model

Keep the learned model when its process ends

Training changes live tensors in one running process. Exiting that process discards those allocations. Reconstructing the architecture with its original seed would recreate initial weights, not the weights produced by training. To use the learned model later, another process needs its actual learned values and enough information to interpret them: the component settings, tokenizer assets and relationships between shared parameters.

Continuing training needs more. AdamW’s moments remember earlier gradients; the next scheduled rate, document choice and random words depend on progress outside the learned weights. We will first preserve a model for independent inference, then preserve the additional state that determines its next update.

Use Chapter 10’s encoder, backbone and tied head, Chapter 13’s accumulated and recomputed updates and Chapter 14’s successful-update schedule as the prerequisites. This chapter preserves those existing computations and their state across process exit.

Save after two updates, then load for CPU inference

Reuse Chapter 13’s text model and documents. The model has vocabulary size $266$, residual width $4$, one attention head, feed-forward width $8$, three decoder blocks and capacity for $8$ positions. Initialization uses seed $39$. There are $29$ distinct learned parameter tensors containing $1572$ scalar values. Input lookup and the output head share one of those tensors.

The example plans four successful logical updates and saves after update $2$. Its producer exits. A new inference process loads the saved model and tokenizer, encodes the prompt ab, keeps beginning-of-sequence token 0 and removes the terminal end-of-sequence token 1. Its prompt IDs are [0,99,100]. CPU f64 greedy generation then produces IDs [108,108,55]. The restored tokenizer decodes them to bytes [106,106,53], the text jj5.

That tiny continuation makes the restored computation observable. It is not evidence of useful language quality. Reopening the same saved-after-two model after other training has continued gives the same inference result: the saved weights are a snapshot, not a live reference to later updates.

Preserve the composition and the shared table

A portable model records which text encoder, causal transformer backbone and text head form the model. It stores their semantic settings, named learned tensors and required tokenizer assets. Its tensor records identify names, element types, shapes and values. A device address or CUDA handle cannot tell a new CPU process what those weights mean, so neither belongs to the learned model’s identity.

The text archive has $31$ tensor entries: $29$ learned tensors, one tensor of numerical constants and one tokenizer asset. The constant tensor uses F64 with shape $[2]$ for RoPE base and RMSNorm epsilon. The tokenizer asset uses U32 with shape $[8,2]$ for the eight ordered merge pairs. These two entries help interpret the learned model; they are not additional learned weights.

The embedding tensor token_embedding.weight is stored once. Its alias record says that the text head uses this same table, transposed for vocabulary projection. On restore, the existing model constructor makes lookup and the head refer to the same live autograd node. Two separately allocated arrays with equal numbers would preserve the first forward result but lose the shared parameter and its combined gradient. The restore tests check both tied paths, not only their initial numerical equality.

A learned model identity hashes normalized semantic settings, the named tensor dtypes, shapes and values, required assets and sharing policy. Optimizer progress and device selection belong to other identities. Matching logical model identity does not promise identical SafeTensors file bytes: metadata ordering can differ while the model’s meaning and values remain the same.

The generic backbone retains its source-independent boundary. Its archive has $28$ learned backbone tensors and one constants tensor, with no vocabulary or tokenizer requirement. The probe supplies existing text-derived embedding vectors and a causal layout; restored hidden features have shape $[1,3,4]$ and matching value bits. This is still a text-derived example. It implements no encoder for images, binary data or another source.

Account for the state that determines the next update

Let $t$ count completed successful logical optimizer updates. Let $S_t$ be the mutable training-job state after those updates; fixed model composition and tokenizer assets are also required to interpret its weights. One next update can be described as

Here $\theta_t$ contains all distinct learned tensors, counting the tied table once. $O_t$ contains AdamW’s moments, beta powers, parameter groups, settings and committed step. $P_t$ contains the saved learning-rate schedule, global clipping limit and recomputation policy.

$D_t$ identifies the data and stores document order, next-document cursor $c_t$ and completed traversal count. $B_{c_t}$ is the next logical batch: in this example, two one-document microbatches selected cyclically from that order. $R_t$ contains dropout policy and next counter plus the independent monitoring-sampling generator state. $L_t$ contains the dynamic loss scale, its bounds, growth interval and finite-update progress. $A_t$ gives the microbatch count per logical update and the pending count.

The transition $F_E$ reuses the existing accumulated trainer and the caller’s data and random advancement after a successful update. Its envelope $E$ records the actual executable’s SHA-256 and the declared same-binary scalar CPU f64 serial execution semantics. Restore also checks the expected data identity. This envelope is an explicit compatibility requirement supplied by the caller, not an automatic proof that all processors or runtime environments perform identical arithmetic.

The equation accounts for state; it introduces no new learning algorithm. In the measured fixture, restoring that state and using compatible execution reproduces the remaining observations and checked final state bits. It does not establish universal bitwise agreement across devices, builds or precisions.

Resume at the next complete logical update

A checkpoint is accepted only with zero pending microbatches and cleared real parameter gradients. These conditions put the saved state between complete updates: there is no partial gradient contribution or active differentiation graph to reconstruct. Capture refuses a nonzero pending count or actual pending gradients; refusal leaves those gradients intact. Mid-update resume is not implemented.

The example uses two one-row microbatches per logical update, block recomputation, dropout probability $0.25$ and global gradient limit $1$. Its four-update schedule has one warmup update, peak rate $0.001$ and final rate $0.0001$. Dynamic loss scaling starts at $8$, stays between $1$ and $32$, and grows after every three finite successful updates.

The fixed document-order array is [2,0,1]. Source indices 0, 1 and 2 name short/full, long/full and medium/full, respectively. The cursor indexes that order array; it is not itself a source-document index or a shuffled iterator. After update $2$, the next cursor is $1$. The next two entries are therefore source indices 0 and 1: short/full, then long/full.

At this saved boundary, one traversal is complete, the dropout next counter is $288$, and scale $8$ has two finite updates toward its growth interval of $3$. The dropout seed is the full-width u64 value 18446744073709551613. The separate sampling generator’s saved state is 4354685564936845350. It chooses one monitoring token after each successful update; that token never feeds back into the training loss. Seeds and counters use standard U64 tensor values, rather than passing through a floating-point number or a decimal string bit parser.

The restored job performs the following remaining updates. Losses are forward means in nats per valid next-token target for each update, before that update’s parameter commit.

Update $3$ completes the third finite update, so it uses scale $8$ and grows the next scale to $16$. It leaves cursor $0$, traversal count $2$ and dropout counter $432$. Update $4$ leaves cursor $2$, traversal count $2$, counter $552$, scale $16$ and one finite update toward the next growth. Both clear gradients. All four scheduled updates are now complete, so no next rate remains.

A separate resume process restores the boundary after update $2$ and executes $3,4$. An independent reference process trains from initialization through $1,2,3,4$. Their remaining-update observations match. Their checked learned weight bits, AdamW moments, beta powers and step, and recorded data, random and scaler progress also match. This is bounded CPU f64 continuation evidence, not a held-out evaluation or a physical-GPU portability result.

A weights-only warm start begins a new training history

Start both paths from the same learned weights saved after update $2$. Full resume retains the checkpoint’s history and applies original updates $3$ and $4$. The weights-only path puts those same learned weights and tokenizer into a fresh job and performs its own updates $1$ and $2$. Each path performs two additional successful updates.

The warm start resets AdamW moments and powers, the rate clock, data cursor, dropout counter, scaler progress and sampling generator together. It first selects medium/full and short/full, rather than the resume path’s short/full and long/full, and begins at rate $0.001$ rather than $0.0003250000000000001$. The final learned weights differ.

This comparison changes several states together. It cannot measure how much one reset alone contributes to the difference. The saved weights remain useful for inference and for a deliberate new training history; they simply do not determine the original job’s next update by themselves.

Preserve neural language-model representations and training memory

Tied neural language-model embeddings serve both input lookup and output projection. Saving two equal arrays would preserve values while losing the shared parameter identity; saving weights alone also omits an optimizer’s training history.

Press and Wolf’s Using the Output Embedding to Improve Language Models recommends tying input and output embeddings in neural language models. The paper was first submitted in August 2016; the cited revision is from February 2017. Its contribution concerns language-model representations and their updates, not the invention of checkpoint formats. The useful persistence consequence is that a tied parameter’s two roles must remain one learned parameter after loading.

Model-saving practice also distinguishes learned model state for inference from optimizer and progress state needed to continue training. The PyTorch Saving and Loading Models tutorial, created in 2018 and updated in 2025, separates model parameters from optimizer state and includes optimizer and epoch state in a general training checkpoint. Those are documentation dates, not invention dates. The tutorial also describes loading GPU-trained model weights on CPU; it does not establish that this course’s CUDA-to-CPU experiment has run.

A reusable decoder must preserve the interpretation and sharing of its learned tensors, while exact continuation must also preserve the state that determines its next update. We preserve the existing encoder, backbone and tied head as one model, then add optimizer, data, random and update-progress state for complete-boundary continuation.

SafeTensors’ shared-tensor documentation explains why arbitrary sharing is not automatically retained as separate named tensor entries. That undated page motivates making sharing explicit; this Rust implementation restores its known alias through the existing model constructor and does not use the page’s Python API.

The runnable contrast starts both paths from the same saved weights and performs two additional updates: full resume retains the training history, while a warm start resets several states together. The measured difference does not isolate one reset as its cause. The course’s exact data order, random state, boundary between complete updates and executable envelope are local choices checked by its Rust, not requirements attributed to those sources.

Reuse model reconstruction and the existing training transition

PortableModel::from_text captures existing named parameters, semantic settings, tokenizer merge pairs and the tied-head alias. SafeTensors owns named-tensor framing and parsing; config-lib owns CONF settings syntax. The adapter copies logical values into the existing tensor owners and makes no zero-copy or speed claim. Standard F64 tensors preserve represented weights and numerical constants, including values whose bits should not be changed by decimal settings serialization.

    /// Preserve the real text composition, including its sole shared token table.
    pub fn from_text(model: &DecoderModel, tokenizer: &BpeTokenizer) -> ArchiveResult<Self> {
        if tokenizer.layout().vocabulary_size() != model.config().vocabulary_size() {
            return Err("saved text tokenizer and model vocabulary differ".into());
        }
        let state = DecoderModelState::snapshot(model);
        let mut config = settings(state.config().block_config(), state.config().layers())?;
        config.set("composition", "text-decoder-v1")?;
        config.set("encoder.kind", "byte-bpe-embedding-v1")?;
        config.set(
            "encoder.vocabulary",
            i64::try_from(state.config().vocabulary_size())?,
        )?;
        config.set("head.kind", "tied-embedding-transpose-v1")?;
        config.set("head.alias", "token_embedding.weight")?;
        config.set(
            "encoder.tokenizer_layout",
            i64::from(tokenizer.layout().version()),
        )?;
        let mut tensors = state
            .named_tensors()
            .map(|(name, tensor)| {
                (
                    name.to_owned(),
                    NamedData::f64(tensor.shape().to_vec(), tensor.as_slice()),
                )
            })
            .collect::<BTreeMap<_, _>>();
        let pairs = tokenizer
            .merge_rules()
            .iter()
            .flat_map(|rule| {
                let pair = rule.training_pair();
                [pair.left(), pair.right()]
                    .into_iter()
                    .flat_map(u32::to_le_bytes)
            })
            .collect();
        tensors.insert(
            MERGES.into(),
            NamedData {
                dtype: Dtype::U32,
                shape: vec![tokenizer.merge_rules().len(), 2],
                bytes: pairs,
            },
        );
        tensors.insert(
            CONSTANTS.into(),
            NamedData::f64(
                vec![2],
                &[state.config().rope_base(), state.config().rms_epsilon()],
            ),
        );
        Ok(Self {
            settings: config,
            tensors,
        })
    }

Restoration delegates learned tensor checks and assembly to existing constructors. The complete text model restores its tokenizer; the generic backbone restores without text assets. The shared backbone assembler builds the existing blocks and final norm for both cases.

    pub fn restore_text(&self) -> ArchiveResult<(DecoderModel, BpeTokenizer)> {
        self.normalized_settings()?;
        if self.composition()? != ModelComposition::TextDecoder {
            return Err("archive is a backbone, not a text model".into());
        }
        let (config, layers) = self.backbone_config()?;
        let merges = self.tensors.get(MERGES).ok_or("missing tokenizer asset")?;
        if merges.dtype != Dtype::U32 || merges.shape.len() != 2 || merges.shape[1] != 2 {
            return Err("invalid text merge asset shape or dtype".into());
        }
        let pairs = merges
            .bytes
            .chunks_exact(8)
            .map(|b| {
                crate::tokenizer::bpe_trainer::TokenPair::new(
                    u32::from_le_bytes(b[..4].try_into().unwrap()),
                    u32::from_le_bytes(b[4..].try_into().unwrap()),
                )
            })
            .collect();
        let saved_tokenizer = CheckpointTokenizer::from_byte_bpe_pairs(pairs)?;
        let vocabulary = integer(&self.settings, "encoder.vocabulary")?;
        if saved_tokenizer.vocabulary_size() != vocabulary {
            return Err("saved text vocabulary and tokenizer asset differ".into());
        }
        let model_config = DecoderModelConfig::new(
            vocabulary,
            config.model_width(),
            config.heads(),
            config.feed_forward_width(),
            layers,
            config.max_positions(),
            config.rope_base(),
            config.rms_epsilon(),
        )
        .with_attention(config.kv_heads(), config.context_window());
        let model = DecoderModel::from_parameters(model_config, self.parameters()?)?;
        let tokenizer = saved_tokenizer
            .restore_bpe()?
            .ok_or("missing byte BPE tokenizer")?;
        Ok((model, tokenizer))
    }

    pub fn restore_backbone(&self) -> ArchiveResult<OwnedDecoderBackbone> {
        self.normalized_settings()?;
        if self.composition()? != ModelComposition::Backbone {
            return Err("archive contains a complete text model".into());
        }
        let (config, layers) = self.backbone_config()?;
        Ok(OwnedDecoderBackbone::from_parameters(
            config,
            layers,
            self.parameters()?,
        )?)
    }

infer receives a portable archive, not a training fixture. It restores both weights and tokenizer, selects no-grad inference and uses the existing greedy generation path on CPU.

/// Only the portable artifact supplies weights and the text tokenizer here.
/// No training fixture, optimizer, CUDA context or resume envelope is required.
pub fn infer(archive: &PortableModel, prompt: &str, new_tokens: usize) -> ArchiveResult<Value> {
    let (model, tokenizer) = archive.restore_text()?;
    let mut ids = tokenizer.encode_utf8_document(prompt);
    ids.pop(); // A generation prompt keeps BOS but removes its terminal EOS.
    let forward = model.forward_with_context(AutogradContext::no_grad(), &ids, &[1, ids.len()])?;
    let logits = forward.logits().value_snapshot();
    let mut rng = SplitMix64::from_seed(39);
    let generated = generate_uncached(
        &model,
        &ids,
        GenerationConfig::new(SamplingMode::Greedy, Some(1), new_tokens),
        &mut rng,
    )?;
    let content = generated
        .generated()
        .iter()
        .copied()
        .filter(|&id| id != 1)
        .collect::<Vec<_>>();
    let decoded = tokenizer.decode_content(&content);
    Ok(
        json!({"model_id":archive.model_id()?,"device":"cpu","precision":"f64", "prompt":prompt,
        "prompt_ids":ids,"generated_ids":generated.generated(),
        "generated_bytes":decoded.as_ref().ok(),"decode_error":decoded.err().map(|e|e.to_string()),
        "logit_bits_sha256":bit_digest(logits.as_slice().iter().map(|x|x.to_bits())),
        "named_parameter_tensors":model.parameters().len(),"parameter_scalars":model.parameter_count(),
        "tied_alias":model.text_output_head().table().is_same_node(model.embedding().table().tensor())}),
    )
}

TrainingCheckpoint extends the existing validated model and AdamW snapshots. It also stores the schedule and update policies, data identity and progress, dropout and sampling state, loss-scaler progress and accumulation settings. The caller still owns the data; the inference model does not embed the dataset. The progress type makes the next document position and random state explicit.

/// The exact next data position and randomness for the caller's training loop.
/// Order lists document indices; cursor points to the next document in that order.
/// Saves in this version require zero pending microbatches and cleared gradients.
#[derive(Clone, Debug)]
pub struct TrainingProgress {
    pub data_sha256: String,
    pub document_order: Vec<usize>,
    pub cursor: usize,
    pub epoch: u64,
    pub accumulation_microbatches: usize,
    pub pending_microbatches: usize,
    pub recompute_blocks: bool,
    pub dropout: DropoutRunPolicy,
    pub dropout_counter: DropoutCounterState,
    pub sampling_rng_state: u64,
}

impl TrainingProgress {
    fn validate(&self) -> ArchiveResult<()> {
        self.dropout.validate()?;
        if self.pending_microbatches != 0 || self.accumulation_microbatches == 0 {
            return Err(
                "checkpoint requires a complete update with zero pending accumulation".into(),
            );
        }
        let mut order = self.document_order.clone();
        order.sort_unstable();
        if order.is_empty()
            || !order.iter().copied().eq(0..order.len())
            || self.cursor >= order.len()
        {
            return Err("invalid training document order or next-document cursor".into());
        }
        if self.dropout.seed != self.dropout_counter.seed() || self.data_sha256.is_empty() {
            return Err("dropout seed or data identity mismatch".into());
        }
        Ok(())
    }
}

The existing accumulated trainer performs differentiation, unscaling, clipping and AdamW. After a successful complete update, the caller advances the fixed document cursor and samples one monitoring token. Restoring the saved dropout seed and counter rebuilds its derived-key cache; future random words need not depend on preserving an internal cache flag.

/// The existing accumulated trainer owns all differentiation, clipping and AdamW.
/// Data and RNG cursors are explicit caller state. This fixture has no overflow.
pub fn advance(job: &mut RestoredTraining, until: usize) -> ArchiveResult<Vec<Value>> {
    let execution = TensorExecution::cpu();
    advance_with_execution(job, until, &execution)
}

/// Later numerical chapters select execution without duplicating this loop.
pub fn advance_with_execution(
    job: &mut RestoredTraining,
    until: usize,
    execution: &TensorExecution,
) -> ArchiveResult<Vec<Value>> {
    let source = examples(Partition::Train)?;
    if job.progress.data_sha256 != data_id()?
        || job.progress.document_order.len() != source.len()
        || !job.progress.recompute_blocks
        || job.progress.pending_microbatches != 0
    {
        return Err("fixture data, recomputation policy, or update boundary differs".into());
    }
    let current = usize::try_from(job.optimizer.step_count())?;
    if until < current || until > job.plan.schedule().steps() {
        return Err("requested update is outside remaining schedule".into());
    }
    let mut updates = Vec::new();
    while job.optimizer.step_count() < until as u64 {
        let step = usize::try_from(job.optimizer.step_count())? + 1;
        let mut selected = Vec::new();
        for offset in 0..job.progress.accumulation_microbatches {
            let index = job.progress.document_order
                [(job.progress.cursor + offset) % job.progress.document_order.len()];
            selected.push(source[index].clone());
        }
        let microbatches = VariableBatchEpoch::build(
            Partition::Train,
            &selected,
            selected[0].binding(),
            VariableBatchConfig::new(1, 8, 8)?,
        )?;
        let outcome = decoder_update_with_accumulation(
            &job.model,
            &mut job.optimizer,
            &job.plan,
            step,
            execution,
            &mut job.scaler,
            &microbatches,
            |context, batch| {
                let forward = job.model.forward_padded_recomputed_with_dropout(
                    context,
                    batch,
                    batch.binding(),
                    DropoutMode::Train,
                    &job.progress.dropout,
                    &mut job.progress.dropout_counter,
                    DropoutLimits::default(),
                )?;
                Ok(batch.loss_with_context(context, forward.logits())?)
            },
            &mut NoDecoderUpdateObserver,
        )?;
        let AccumulatedDecoderUpdate::Applied {
            execution: applied,
            microbatches: count,
            valid_target_count,
            loss_scale,
            next_loss_scale,
        } = outcome
        else {
            return Err(
                "bounded finite fixture unexpectedly overflowed; no resume equivalence claimed"
                    .into(),
            );
        };
        let advanced = job.progress.cursor + count;
        job.progress.epoch += u64::try_from(advanced / job.progress.document_order.len())?;
        job.progress.cursor = advanced % job.progress.document_order.len();
        // Sample one local monitoring continuation after each complete update.
        // Persist the advanced RNG; generated text does not enter the training loss.
        let mut sampling = SplitMix64::from_state(job.progress.sampling_rng_state);
        let monitoring = generate_uncached(
            &job.model,
            &[0, 99],
            GenerationConfig::new(
                SamplingMode::TemperatureTopK {
                    temperature: 1.,
                    top_k: 8,
                },
                Some(1),
                1,
            ),
            &mut sampling,
        )?;
        job.progress.sampling_rng_state = sampling.state();
        updates.push(json!({"step":step,"learning_rate":applied.step.learning_rate(),
            "mean_loss":applied.step.train_loss(),"microbatches":count,"valid_targets":valid_target_count,
            "loss_scale":loss_scale,"next_loss_scale":next_loss_scale,
            "data_document_ids":selected.iter().map(|e|e.id()).collect::<Vec<_>>(),
            "next_cursor":job.progress.cursor,"epoch":job.progress.epoch,
            "dropout_next_counter":job.progress.dropout_counter.next_counter(),
            "monitoring_token_ids":monitoring.generated(),"sampling_rng_state":sampling.state(),
            "gradients_cleared":applied.step.cleared_gradients()}));
    }
    Ok(updates)
}

The warm-start comparison replaces only the fresh job’s model and tokenizer with the saved model, leaving its other state fresh. Its two updates therefore belong to a new training clock.

/// A usable saved inference model is not by itself an exact training checkpoint.
/// This warm start deliberately resets the optimizer, schedule and data/RNG state.
fn weights_only_restart(
    archive: &PortableModel,
    continued: &RestoredTraining,
) -> ArchiveResult<Value> {
    let mut restarted = fresh_job()?;
    (restarted.model, restarted.tokenizer) = archive.restore_text()?;
    let updates = advance(&mut restarted, TOTAL_UPDATES - CUT_AFTER)?;
    let changed = DecoderModelState::snapshot(&restarted.model).bit_pattern()
        != DecoderModelState::snapshot(&continued.model).bit_pattern();
    Ok(
        json!({"restart_optimizer_step":restarted.optimizer.step_count(),
        "continuation_optimizer_step":continued.optimizer.step_count(),
        "additional_updates_per_path":TOTAL_UPDATES-CUT_AFTER,
        "restarted_updates":updates,"weights_differ_from_continuation":changed,
        "reset_state":["AdamW moments and powers","learning-rate clock","data cursor","dropout counter","loss-scale progress","sampling RNG"]}),
    )
}

Restore refuses changed data identity or a changed resume envelope before returning mutable training state. The model and training files use the existing atomic replacement helper. Their checked corruption cases are refused; those checks are not an authenticity guarantee or a claim about every hostile file.

Portable model loading has no mandatory CUDA backend or old device handle. CPU inference needs no NVIDIA context or retraining. Training on physical CUDA, saving, exiting and loading the learned weights on CPU remains a separate pending hardware experiment.

Follow the saved model and its two uses

The figure joins the two restored table uses to one parameter identity. It also separates the model required by inference from the additional history required by the original training job. The displayed generated IDs and update numbers come from the observed Rust fixture.

Reproduce the process boundary and inspect the saved state

The existing clap CLI selects train, infer or resume. Without a subcommand it prints the bounded in-memory round trip and warm-start comparison.

use clap::{Parser, Subcommand};
use llm_from_scratch_practical::checkpoint::{portable::PortableModel, resume::TrainingCheckpoint};
use practical_ch15_save_load_resume_model::{
    advance, data_id, demo, executable_envelope, fresh_job, infer, save_job, state_record,
};
use serde_json::json;
use std::path::PathBuf;

#[derive(Parser)]
#[command(about = "Save a bounded text model, load it for CPU inference, or resume training")]
struct Cli {
    #[command(subcommand)]
    command: Option<Command>,
}

#[derive(Subcommand)]
enum Command {
    Train {
        #[arg(long,default_value_t=2,value_parser=clap::value_parser!(u32).range(1..=4))]
        until: u32,
        #[arg(long)]
        model: PathBuf,
        #[arg(long)]
        checkpoint: PathBuf,
    },
    Infer {
        #[arg(long)]
        model: PathBuf,
        #[arg(long, default_value = "ab")]
        prompt: String,
        #[arg(long,default_value_t=3,value_parser=clap::value_parser!(u32).range(1..=7))]
        tokens: u32,
    },
    Resume {
        #[arg(long)]
        checkpoint: PathBuf,
        #[arg(long,default_value_t=4,value_parser=clap::value_parser!(u32).range(1..=4))]
        until: u32,
        #[arg(long)]
        model: PathBuf,
        #[arg(long)]
        save: PathBuf,
    },
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();
    let result = match cli.command {
        None => demo()?,
        Some(Command::Infer {
            model,
            prompt,
            tokens,
        }) => infer(&PortableModel::load(model)?, &prompt, tokens as usize)?,
        Some(Command::Train {
            until,
            model,
            checkpoint,
        }) => {
            let envelope = executable_envelope()?;
            let mut job = fresh_job()?;
            let updates = advance(&mut job, until as usize)?;
            save_job(&job, &envelope, &model, &checkpoint)?;
            json!({"updates":updates,"state":state_record(&job)?,"model_saved":true,"training_checkpoint_saved":true})
        }
        Some(Command::Resume {
            checkpoint,
            until,
            model,
            save,
        }) => {
            let envelope = executable_envelope()?;
            let saved = TrainingCheckpoint::load(checkpoint)?;
            let mut job = saved.restore(&envelope, &data_id()?)?;
            let updates = advance(&mut job, until as usize)?;
            save_job(&job, &envelope, &model, &save)?;
            json!({"updates":updates,"state":state_record(&job)?,"model_saved":true,"training_checkpoint_saved":true})
        }
    };
    println!("{}", serde_json::to_string_pretty(&result)?);
    Ok(())
}

From the repository root, with Docker and the existing course Docker setup available, run the default CPU example. No GPU, external corpus or downloaded model is needed:

COURSE_CORPUS=false ./course run cargo run --quiet --offline --locked -p practical-ch15-save-load-resume-model

The JSON reports portable_inference, boundary_state, resumed_updates, final_state and weights_only_restart. The four continuation comparisons weight_bits_equal, optimizer_bits_equal, remaining_updates_equal and progress_equal are true. The weights-only comparison reports two additional updates per path and weights_differ_from_continuation is true. This command alone uses an in-memory round trip; it does not demonstrate exit and loading in another process.

For independent processes, keep the same repository-root working directory and Docker setup and run this entire block as one command. Each Cargo invocation runs a program that exits before the next starts. The one container keeps their saved files available:

COURSE_CORPUS=false ./course run bash -euc '
mkdir -p tmp/ch15
cargo run --quiet --offline --locked -p practical-ch15-save-load-resume-model -- train --until 2 --model tmp/ch15/after2.safetensors --checkpoint tmp/ch15/after2-training.safetensors
cargo run --quiet --offline --locked -p practical-ch15-save-load-resume-model -- infer --model tmp/ch15/after2.safetensors --prompt ab --tokens 3
cargo run --quiet --offline --locked -p practical-ch15-save-load-resume-model -- resume --checkpoint tmp/ch15/after2-training.safetensors --until 4 --model tmp/ch15/resumed4.safetensors --save tmp/ch15/resumed4-training.safetensors
cargo run --quiet --offline --locked -p practical-ch15-save-load-resume-model -- train --until 4 --model tmp/ch15/full4.safetensors --checkpoint tmp/ch15/full4-training.safetensors
cargo run --quiet --offline --locked -p practical-ch15-save-load-resume-model -- infer --model tmp/ch15/resumed4.safetensors --prompt ab --tokens 3
cargo run --quiet --offline --locked -p practical-ch15-save-load-resume-model -- infer --model tmp/ch15/full4.safetensors --prompt ab --tokens 3
'

The first inference record contains generated IDs [108,108,55], bytes [106,106,53] and tied_alias: true. The resume record’s updates array contains steps 3 and 4; the independent train record contains 1 through 4. Their final state objects match, including model and optimizer digests, cursor, random state and scaler progress. The final two inference records also match. The wrapper uses a disposable container: separate ./course run invocations would not retain these files.

With the same repository-root working directory and Docker setup, run the five demo library tests:

COURSE_CORPUS=false ./course run cargo test --offline --locked -p practical-ch15-save-load-resume-model

The library result is five passed tests. They include continuation and refusals, a real restored tied alias, corruption checks, grouped optimizer state before the first update and pending-gradient refusal without clearing the gradients.

Optional inspection and explanation practice

1. Use the default command above to inspect the portable model and checkpoint roles. Explain why the archive’s $31$ entries are not $31$ learned tensors.
2. Use the complete single-container command above to inspect saving, producer exit, new-process inference and resumed training. Compare the two final state objects and the final two inference records.
3. Use the saved order [2,0,1], cursor $1$ and finite-scale progress $2$ toward interval $3$ to explain the documents and scale used by update $3$.
4. Inspect weights_only_restart in the default command’s output. Explain why both paths perform two more updates while their final optimizer counts and weights differ.
5. Use the five-test command above and the default record’s refusal fields to inspect the complete-update boundary and compatibility limits.

Evaluate the model that can now be loaded independently

The same text encoder, source-independent backbone and tied head now survive process exit with their learned weights and required text assets. CPU inference loads that model independently. A full training checkpoint also restores the next complete update under the declared data and execution conditions.

Planned Chapter 16 will evaluate the saved language model and bound conclusions about its quality. Physical CUDA training followed by CPU loading, full-corpus training and accelerated mixed-precision execution remain separate pending evidence. The small restored continuation here establishes persistence and state consistency, not broad language quality or training throughput.