Keep model operations and failures explicit

The problem: failure leaves different kinds of state

A decoder can refuse invalid dimensions before construction, fail during an optimizer update, or fail while publishing a checkpoint. These operations touch different state. A caller needs to know which operation failed, which state remains usable, and whether unfinished work still needs cleanup. An error message alone does not establish those facts.

This chapter follows the existing model operations and attaches typed failures to their precise boundaries. Start with configuration: settings can be readable while describing a model that cannot divide its residual features into heads.

Decode settings, then validate the model

The shipped decoder-profiles.conf supplies a complete reference model with residual width $4$ and one query head. Each query head therefore has $4$ features. F14 keeps the other reference fields but changes the native settings to d = 5 and hq = 2. The settings syntax loads, but the existing ModelConfig refuses the model: five features cannot be divided evenly between two query heads. The operation does not use a fractional or truncated quotient as a tensor dimension.

The existing path reads native CONF syntax through Config::from_string, then passes the loaded values to ModelConfig::from_config:

let settings = Config::from_string(source, Some("conf"))?;
let model = ModelConfig::from_config(&settings)?;

The configuration library reads syntax and checks general field types and presence. The existing course validator checks the decoder’s dimensions, head grouping, policies and identities. This is the ModelConfig already used in Chapter 7.

Recall the head-width invariant

For positive integer residual width and query-head count, exact divisibility must hold before computing the head width:

Name the model dimensions

$d$

Residual width: the number of features per token.

$h_q$

The number of query heads that share those residual features.

$d_{\mathrm{head}}$

The number of features in each query head.

The reference has $d=4$ and $h_q=1$, so $d_{\mathrm{head}}=4$. F14’s $d=5$ and $h_q=2$ violate the divisibility condition. A zero head count is also refused before division. The full existing validator additionally requires a positive key/value-head count that divides the query-head count, an even head width for RoPE, and positive remaining model dimensions. It checks the supported model and dtype policies, numerical constants, and vocabulary and tokenizer identities. The head-width formula recalls one invariant; it is not the complete model validator.

Unknown and duplicate keys do not have a strict-rejection policy in this native path. A duplicate can change the resulting model descriptor. Use the actual loaded values and their identity when admitting a configuration. ConfigError::Invalid is deliberately coarse: the boundary maps it to config/invalid_value and retains the typed source. It does not parse an English message to invent a finer divisibility or field-error token.

These APIs return descriptors and receive no live decoder, dropout counter or cache owner. Reading or refusing settings cannot itself publish a model or advance those owners.

Read failure identity and state effect separately

F14’s category is config, its reason is invalid_value, and its phase is config_admission. Its terminal state is rejected. Its disposition is reject_unchanged: configuration admission refused the candidate without changing the declared live state. The reason identifies the failed contract; the disposition states the allowed continuation.

FunctionalError wraps existing component errors. Its live Error::source retains the original typed cause where one exists. The public diagnostic record contains stable tokens and bounded requested/actual identity digests, with a maximum encoded size of $4096$ bytes. If selection never occurred, actual identity is not_selected. Copying the request into that field would falsely claim that the requested implementation was selected.

impl FunctionalError {
    pub fn new(
        reason: FailureReason,
        phase: FailurePhase,
        disposition: FailureDisposition,
        identity: OperationIdentity,
    ) -> FunctionalResult<Self> {
        if !valid_tuple(reason, phase, disposition) {
            return Err(Self::record_rejection(
                FailureReason::SchemaInvalidTuple,
                identity,
            ));
        }
        let terminal = match reason {
            FailureReason::Cancellation => TerminalState::Cancelled,
            FailureReason::Deadline => TerminalState::TimedOut,
            _ => match phase {
                FailurePhase::Execution | FailurePhase::Commit | FailurePhase::Cleanup => {
                    TerminalState::Failed
                }
                _ => TerminalState::Rejected,
            },
        };
        Ok(Self {
            reason,
            phase,
            disposition,
            terminal,
            identity,
            source: None,
            original: None,
        })
    }
    fn record_rejection(reason: FailureReason, identity: OperationIdentity) -> Self {
        Self {
            reason,
            phase: FailurePhase::ConfigAdmission,
            disposition: FailureDisposition::RejectUnchanged,
            terminal: TerminalState::Rejected,
            identity,
            source: None,
            original: None,
        }
    }
    pub const fn reason(&self) -> FailureReason {
        self.reason
    }
    pub const fn category(&self) -> FailureCategory {
        self.reason.category()
    }
    pub const fn phase(&self) -> FailurePhase {
        self.phase
    }
    pub const fn disposition(&self) -> FailureDisposition {
        self.disposition
    }
    pub const fn terminal_state(&self) -> TerminalState {
        self.terminal
    }
    pub fn identity(&self) -> &OperationIdentity {
        &self.identity
    }
    pub fn with_source(mut self, source: impl Error + 'static) -> Self {
        self.source = Some(Box::new(source));
        self
    }
    pub fn original_failure(&self) -> Option<&FunctionalError> {
        self.original.as_deref()
    }
    pub fn record(&self) -> FailureRecord {
        FailureRecord {
            schema_version: 1,
            category: self.category().token().into(),
            reason: self.reason.token().into(),
            phase: self.phase,
            disposition: self.disposition,
            terminal_state: self.terminal,
            identity: self.identity.clone(),
        }
    }
    pub fn canonical_record(&self) -> Vec<u8> {
        canonical_json_bytes(&self.record())
            .expect("fixed bounded diagnostic contains no nonfinite values")
    }
    /// Rejections use the caller's trusted identity, never unvalidated wire fields.
    pub fn from_record(
        bytes: &[u8],
        rejection_identity: &OperationIdentity,
    ) -> FunctionalResult<Self> {
        let record: FailureRecord =
            read_canonical_record(bytes, DIAGNOSTIC_BYTES_MAX).map_err(|source| {
                Self::record_rejection(
                    FailureReason::SchemaRecordFields,
                    rejection_identity.clone(),
                )
                .with_source(source)
            })?;
        if record.schema_version != 1 {
            return Err(Self::record_rejection(
                FailureReason::SchemaVersion,
                rejection_identity.clone(),
            ));
        }
        if !FailureReason::ALL
            .iter()
            .any(|r| r.category().token() == record.category)
        {
            return Err(Self::record_rejection(
                FailureReason::SchemaUnknownCategory,
                rejection_identity.clone(),
            ));
        }
        let Some(reason) = FailureReason::from_tokens(&record.category, &record.reason) else {
            return Err(Self::record_rejection(
                FailureReason::SchemaInvalidTuple,
                rejection_identity.clone(),
            ));
        };
        let error = Self::new(reason, record.phase, record.disposition, record.identity).map_err(
            |source| {
                Self::record_rejection(
                    FailureReason::SchemaInvalidTuple,
                    rejection_identity.clone(),
                )
                .with_source(source)
            },
        )?;
        if error.terminal != record.terminal_state {
            return Err(Self::record_rejection(
                FailureReason::SchemaInvalidTuple,
                rejection_identity.clone(),
            ));
        }
        Ok(error)
    }
}

The diagnostic decoder rejects unknown fields, unknown categories, unsupported versions and impossible combinations of reason, phase, disposition and terminal state. For example, changing F27’s rejected admission record to claim terminal state failed produces schema/invalid_error_tuple. An invalid record’s identity is untrusted, so the refusal uses the caller’s trusted identity. Serializing a failure preserves its public diagnostic fields; reading that record cannot recreate the original private live source object.

An unexpected supporting error becomes internal/unmapped_source with its source retained and disposition stop_job. Ordinary successful generation stops remain success. No automatic retry or substitute follows a failure.

Name the local snapshot and commit point

The local transition harness starts with model revision 7, dropout next counter 192, cache logical length 3 and published-output count 0. These are four explicit harness counters. They are not a snapshot of a real decoder, cache or random generator. The harness owns one provisional Vec payload of sixteen bytes and assigns candidate counters only after checked work succeeds.

Injected clock values and explicit barriers make these results deterministic. Cancellation is checked before an already-expired entry deadline. F31 and F32 exercise the corresponding deadline boundaries, producing timed_out.

The successful commit control advances revision to 8 and published count to 1, retaining dropout counter 192 and cache length 3. Cancellation observed by the next operation leaves that committed result intact and performs no second allocation or commit. A later cancellation cannot retract completed work.

/// This chapter-local state machine tests ownership; it is not a model trainer.
fn checked_transition(
    state: &mut State,
    owned: &mut Provisional,
    control: &Control,
) -> Result<(), FunctionalError> {
    use FailureDisposition::*;
    use FailurePhase::*;
    use FailureReason::*;
    if control.cancelled_at_entry {
        return Err(error(Cancellation, RequestAdmission, RejectUnchanged));
    }
    if expired(control.entry_clock) {
        return Err(error(Deadline, RequestAdmission, RejectUnchanged));
    }
    owned
        .allocate()
        .map_err(|source| FunctionalError::unmapped(source, Execution, identity()))?;
    let rejection = if control.execution_failed {
        Some(FunctionalError::unmapped(
            io::Error::other("private execution source"),
            Execution,
            identity(),
        ))
    } else if control.migration_failed {
        Some(error(MigrationFailed, Execution, AbortProvisional))
    } else if control.cancelled_at_barrier {
        Some(error(Cancellation, Execution, AbortProvisional))
    } else if expired(control.barrier_clock) {
        Some(error(Deadline, Execution, AbortProvisional))
    } else if control.evaluation_unavailable {
        Some(error(EvaluationUnavailable, Execution, StopJob))
    } else {
        None
    };
    if let Some(original) = rejection {
        if control.cleanup_failed {
            return Err(FunctionalError::cleanup_failed(
                original,
                io::Error::other("private cleanup source"),
            ));
        }
        owned.discard();
        return Err(original);
    }
    let mut candidate = state.clone();
    candidate.revision = candidate
        .revision
        .checked_add(1)
        .expect("bounded fixture revision");
    candidate.published = candidate
        .published
        .checked_add(1)
        .expect("bounded fixture publication count");
    *state = candidate;
    owned.discard();
    Ok(())
}

F44 retains both the original execution failure and the cleanup cause. The unchanged committed counters do not make the session safe to reuse: cleanup failed, sixteen payload bytes remain owned at the observation boundary, and reuse_allowed is false. The ownership counters measure the fixture’s Vec payload capacity, allocation, release and high-water payload. They do not measure physical allocator retention or process resident memory.

Keep each component’s own preservation boundary

F36 calls the existing AdamW after a backward call supplies the finite gradient f64::MAX to a one-value parameter. Squaring that gradient overflows in the staged optimizer arithmetic. The failure is numeric/nonfinite_optimizer_arithmetic, phase execution, terminal failed, with disposition reject_unchanged. The snapshot is after gradient calculation and before update. Parameter-node identity, parameter and gradient bits, and optimizer state stay unchanged; the optimizer step remains 0. This guarantee does not roll back the preceding gradient calculation or any earlier update.

F43 calls the existing Checkpoint::save_atomic. On the tested Unix filesystem boundary, a rename cannot replace a nonempty owned directory. The failure is artifact/publication_failed, phase commit, terminal failed, with disposition abort_provisional. The last-good file bytes remain intact and no temporary successor remains. This observation concerns that publication operation; it does not establish recovery on every filesystem or database. The checkpoint’s corruption checks retain its existing non-cryptographic checksum and do not establish full artifact SHA-256 admission.

Chapter 8’s complete-forward guarantee likewise preserves its live dropout counter on a failed forward. A later training update has a separate boundary. These results do not supply whole-job rollback or durable restart.

Recall the model operation behind the configuration

A basic recurrent encoder-decoder can compress the source into one fixed-size vector. In their 2014 paper, Bahdanau, Cho and Bengio compute a new context as a softmax-weighted sum of encoder annotations scored against the preceding decoder state. The later term additive attention describes that scoring approach; the decoder still advances recurrently.

The Transformer forms scaled query-key dot products for simultaneous queries, normalizes each query row, and uses those weights to mix values. Its self-attention relates positions within one sequence. This is the mechanism described by Vaswani and colleagues in 2017. Forming that score grid together within one layer does not make its work constant or remove autoregressive generation order.

The related Rust comparison calls the existing first-course and practical self-attention functions. Both receive query and key tensors of shape $[1,2,2]$ filled with zeros: one batch, two token positions and two features. The value rows are $[2,0]$ and $[0,4]$. Every query-key score is zero, so each query gives both available positions weight $1/2$. Each output row is therefore $[1,2]$. The executed flat output is [1.0,2.0,1.0,2.0] in both APIs, with probabilities [0.5,0.5,0.5,0.5].

/// Compare the already-taught attention APIs on the same bounded input.
pub fn attention_contrast() -> DemoResult<Value> {
    use llm_from_scratch as reference;
    use llm_from_scratch_practical as practical;
    let q = practical::autograd::tensor_core::TensorValue::constant(
        practical::tensor::storage::Tensor::from_vec(vec![1, 2, 2], vec![0.0; 4])?,
    )?;
    let v = practical::autograd::tensor_core::TensorValue::constant(
        practical::tensor::storage::Tensor::from_vec(vec![1, 2, 2], vec![2.0, 0.0, 0.0, 4.0])?,
    )?;
    let current =
        practical::attention::self_attention::scaled_dot_product_self_attention(&q, &q, &v)?;
    let earlier_q = reference::autograd::tensor_core::TensorValue::constant(
        reference::tensor::storage::Tensor::from_vec(vec![1, 2, 2], vec![0.0; 4])?,
    )?;
    let earlier_v = reference::autograd::tensor_core::TensorValue::constant(
        reference::tensor::storage::Tensor::from_vec(vec![1, 2, 2], vec![2.0, 0.0, 0.0, 4.0])?,
    )?;
    let earlier = reference::attention::self_attention::scaled_dot_product_self_attention(
        &earlier_q, &earlier_q, &earlier_v,
    )?;
    let output = current.output().value().as_slice().to_vec();
    assert_eq!(output, vec![1.0, 2.0, 1.0, 2.0]);
    assert_eq!(output, earlier.output().value().as_slice());
    Ok(
        json!({"query_and_key":[0,0,0,0],"value":[2,0,0,4],"shape":[1,2,2],
        "probabilities":current.probabilities().value().as_slice(),"output":output,
        "same_reference_output":true,"operation":"existing-course-owned-unmasked-attention",
        "scope":"one f64 input; no full-model or quality equivalence"}),
    )
}

The decoder’s attention, loss, gradients and update decisions remain visible in the existing course Rust operations. This is one f64 course-sequence comparison. It does not implement the 2014 mechanism or establish full-model, GPU or quality equivalence.

Read each result at its tested boundary

The tables keep each configuration change, failure boundary and state effect beside its observed result. A diagram would add no relationship beyond those comparisons. The full output contains fifty-two distinct failures, F01 through F52. Positive controls are additional and do not inflate that count.

The local transition cases test their explicit counters and owned payload. The future-boundary cases do not execute GPU kernels, a server, database migration or a request-page allocator. An availability refusal in this harness does not establish integration with those components.

The compiled dev-profile compact fixture took about $9.48$ seconds with peak child resident set size $14\,823\,424$ bytes; the full-row fixture took about $4.82$ seconds with $14\,684\,160$ bytes. Each ran with one CPU and a $256$ MiB memory cap. These CPU-child measurements exclude compilation, container infrastructure and the measurement wrapper. They do not measure a future backend or physical allocator retention.

Optional practice: reproduce and explain the boundaries

Run from the repository root with Docker running. The existing course launcher prepares the workspace image, whose first build may need network access to provision the toolchain and dependencies. Set COURSE_CORPUS=false as shown to disable corpus-asset acquisition; these supplied CPU fixtures need no corpus or model download. After setup, the command runs locked offline Cargo with container networking disabled. The compact command uses the default dev profile and prints one JSON object for 09-dependency-error-contract: faults.total is 52, positive_controls.native_head_width is 4, and attention.same_reference_output is true.

COURSE_CORPUS=false ./course run cargo run --offline --locked -p practical-ch09-dependency-error-contract

From the same working directory and setup, print all fifty-two newline-delimited JSON failure records, F01 through F52, with the following command. Each row contains its diagnostic, scope, before/after counters and ownership observation. F44 reports reuse_allowed: false and sixteen live payload bytes; F36 reports optimizer step zero and snapshot phase after-gradient-before-update.

COURSE_CORPUS=false ./course run cargo run --offline --locked -p practical-ch09-dependency-error-contract --example practical-ch09-dependency-error-contract-faults

The focused checks use the same repository-root Docker setup. The first command selects the operation-error integration target and reports six passing tests. The second runs the demo library’s two tests, including all fifty-two distinct injections.

COURSE_CORPUS=false ./course run cargo test --offline --locked -p llm-from-scratch-practical --test practical_ch09_dependency_error_contract
COURSE_CORPUS=false ./course run cargo test --offline --locked -p practical-ch09-dependency-error-contract --lib

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let output = practical_ch09_dependency_error_contract::run()?;
    let bytes = llm_from_scratch_practical::artifact_identity::canonical_json_bytes(&output)?;
    print!("{}", std::str::from_utf8(&bytes)?);
    Ok(())
}

Use the compact and full-output commands above for the six optional tasks below.

1. Inspect F14 and the native configuration table. Explain why readable CONF settings can still describe an invalid model, and identify the boundary’s stable reason.
2. Compare F33, F34 and the after-commit cancellation control. Identify the preserved counters, the difference in provisional ownership and the completed work that later cancellation cannot retract.
3. Inspect F36. Identify when its preservation snapshot was taken, explain which arithmetic fails, and distinguish the failed update from the preceding gradient calculation.
4. Inspect F43. Describe the failed checkpoint publication, the last-good and temporary-file results, and the filesystem scope of that observation.
5. Inspect F44. Explain why unchanged committed counters do not permit reuse, and identify the retained payload and failure causes.
6. Inspect F46. Identify the refusal when the requested device is unavailable and a CPU substitute is proposed; explain what would be needed to select another configuration.

Carry explicit failures into input encoding

Configuration refusal, optimizer preservation, checkpoint publication and local cleanup each have a stated boundary. Chapter 10 will carry those typed failures and state identities into the required separation of input encoding from the shared transformer. The encoder and transformer will use them to distinguish malformed shapes or layouts from incompatible live state and reject invalid inputs or state at the appropriate boundary. The text encoder will supply the checked sequence of embedding vectors consumed by the transformer and keep text-specific validation in the encoder.

GPU tensor execution will follow in Chapter 11. It will reuse typed categories, dispositions and requested/actual backend identities to keep device selection explicit. If the requested CUDA backend is unavailable, the operation will refuse explicitly instead of silently substituting a CPU run. The reported actual identity will reflect what was selected, including no selection.

Model settings will describe architecture, tensor shapes and numerical semantics; execution settings will select the device, backend and supported dtype. Keeping those choices separate without changing the learned model’s identity will be a requirement for both future lessons.

These are future implementation requirements. This chapter’s bounded CPU boundary fixtures do not demonstrate the input-encoding boundary, CUDA execution or portable saved tensors. The input boundary, real CUDA integration, activation scheduling and durable job continuation will each need their own evidence.