Store weights in four bits and measure the error

Spend fewer saved bytes on each weight

The model saved in Chapter 15 stores each learned weight as an f64, using $8$ bytes per value. Many nearby weight values can instead share a small set of representable levels. Saving an integer level number and a scale can reduce the weight payload, but reconstructing a level generally changes the weight. Those changes can alter matrix products, token probabilities and generated tokens. A useful comparison therefore needs both a storage count and a measure of what the reconstructed model does.

Weight quantization makes this approximation explicit. We will store each weight as a signed four-bit code, keep a positive scale for a small group of weights, and multiply the code by that scale when loading the model. The existing f64 matrix multiplication and model remain our execution path.

Start with this constructed matrix and input row:

$W$ has shape $[2,4]$, $x$ has shape $[1,2]$, and the product $y=xW$ has shape $[1,4]$. These values are an explanatory fixture, separate from the learned model we will compare later.

Let one scale cover each row

A group size $g=4$ places each complete row in one group. The largest absolute weight in either row is $7$. Choosing scale $s=7/7=1$ makes code $7$ represent weight $7$ and code $-7$ represent weight $-7$.

Divide each weight by its scale and round to the nearest integer. Our tie rule rounds halfway values away from zero: $-3.5$ becomes $-4$, and $3.5$ becomes $4$. In the second row, $0.1$, $0.2$ and $0.3$ all round to zero because the scale is still $1$. The resulting signed codes and reconstructed matrix are

The equal code and weight values here follow from scales of $1$; codes do not generally have the same numerical value as reconstructed weights. In particular, sharing a scale with the large weight $7$ erased the three small values in the second row.

Give smaller groups their own scales

For $g=2$, split each row into its first two and last two entries. The four scales, in row order, are ideally

The first row’s last group now represents $3.5$ exactly as code $7$ times scale $0.5$. The second row’s first group no longer shares a scale with $7$: $0.1/(0.2/7)=3.5$ rounds to code $4$, while $0.2$ uses code $7$. Its last group still contains $0.3$ and $7$, so $0.3$ still becomes zero. The codes are

The actual Rust report prints the third scale as 0.028571428571428574 and the reconstruction of the source value 0.1 as 0.1142857142857143. These are stored f64 observations. The ideal expression $4(0.2/7)$ explains the intended reconstruction; its decimal appearance does not make all the intermediate operations exact in binary floating point.

Fit, round and reconstruct within each row group

For a nonempty matrix with $R$ rows and $C$ columns, choose a positive integer group size $g$. Partition every row into contiguous groups of at most $g$ weights. A final shorter group keeps its own scale; groups never cross rows. For the index set $H$ of one such group, use

$w_i$ is source weight $i$ in the group; $a$ is the group’s largest absolute weight. The scale $s$ gives the weight change represented by one integer code step. Dividing by $s$ converts a weight to code units; rounding selects the nearest level, and clamping restricts the result to $-7$ through $7$. $q_i$ is the dimensionless signed code, and $\widehat w_i$ is its reconstructed weight in the original weight units. The subscript “away” specifies that a halfway value moves away from zero.

This is a symmetric representation with zero point $0$: code $0$ reconstructs weight $0$. It uses $15$ levels, from $-7$ through $7$. Four signed bits could also encode $-8$, but this course representation reserves and rejects that code. An all-zero group uses $s=1$: the positive fallback keeps division defined and gives only zero codes and zero reconstructions. It does not preserve the sign of a floating-point zero.

Separate rounding error from clipping error

In ideal real arithmetic, suppose $s>0$ and the ratio $w_i/s$ lies within $[-7,7]$, so clamping does not change the rounded result. Nearest-integer rounding then gives

Multiplication by the positive scale converts a half-code-step distance to a half-scale weight error. The away-from-zero tie rule attains that bound at either halfway value. Fitting the scale from the absolute maximum keeps the ideal source ratios inside the code range.

A supplied fixed scale is a different situation. At scale $1$, halfway values $0.5$ and $-0.5$ become codes $1$ and $-1$. With $s=1$, source weight $10$ rounds beyond the range and clips to code $7$; reconstruction is $7$ and absolute error is $3$. Source weight $-10$ similarly becomes $-7$. The half-scale bound does not apply to these clipped values. The scalar Rust helper supports this explicit fixed-scale contrast; it is separate from fitting an absolute-maximum scale to the group.

The implementation also rounds during f64 scale fitting, division and reconstruction. The ideal inequality is therefore not a universal exact f64 guarantee. Nonfinite weights, nonpositive or nonfinite scales, and nonfinite reconstructions are refused. Even a positive finite absolute maximum can be too small: dividing it by $7$ can underflow to zero, which is an invalid scale.

Pack two signed codes into one byte

Keep the codes in row-major order. Encode each signed code as a four-bit two’s-complement nibble, put the first code in the low four bits of a byte, and put the next code in the high four bits. For example, code $-7$ has nibble 9, and code $-4$ has nibble C. Their byte is C9: the low nibble still belongs to the first weight.

The $g=4$ code stream [-7,-4,0,4,0,0,0,7] packs as C9 40 00 70. For $g=2$, [-7,-4,0,7,4,7,0,7] packs as C9 70 74 70. These hexadecimal bytes are storage literals, not floating-point values. Nibble 8 would decode as the reserved code $-8$ and is rejected.

Packing follows the continuous code stream, so one byte may contain the last code of one row and the first code of the next. Scale lookup still uses each code’s row and column. An odd total code count leaves the final high nibble unused and zero. For the one-row fixture [-7,-3.5,0,3.5,7] with $g=4$, the last weight forms a shorter second group; scales are $[1,1]$ and packed bytes are C9 40 07. The high half of 07 is padding, not another weight.

Measure the error where the weights are used

For the constructed matrix, the existing matrix product gives

After reconstruction, the same Rust multiplication prints [-7.0,-4.0,0.0,18.0] for $g=4$, and [-6.771428571428571,-3.6,0.0,17.5] for $g=2$. The following summaries compare each reconstructed value with its source:

Mean squared error is $\mathrm{MSE}=M^{-1}\sum_{k=1}^{M}(\widehat z_k-z_k)^2$: $z_k$ and $\widehat z_k$ are the source and reconstructed values being compared, and $M$ counts those values. Here $M=8$ for weights and $M=4$ for product outputs. MSE uses squared weight or output units; maximum absolute error uses the corresponding original units. Error summaries are rounded to twelve decimal places after calculation, while the report retains the normal f64 printing of scales and individual reconstructed values.

The connection to a product is explicit in ideal arithmetic. Let $E=\widehat W-W$ be the matrix of weight errors. Then

Output index $j$ selects one column; input coordinate $i$ weights the error in row $i$ of that column. The triangle inequality gives the bound by adding those absolute contributions. This local linear result explains why weight error can change a layer’s output. It does not bound the quality of an entire transformer. Smaller groups improve the MSE in this fixture, but do not prove that every value or language-model score improves for every group size.

Count scales and complete files as well as codes

For one matrix, the number of weights is $N=RC$, and the number of row groups is $G=R\lceil C/g\rceil$. The original f64 weight payload uses $8N$ bytes. Packed codes use $\lceil N/2\rceil$ bytes, and the f64 scales add $8G$ bytes. The tiny matrix therefore uses $4+32=36$ code-and-scale bytes for $g=2$, or $4+16=20$ for $g=4$, compared with $64$ original weight bytes. These subtotals omit logical shape, other tensors, configuration and the file header.

Each independently stored matrix has its own final padding nibble. For many matrices, add their individual code-byte counts $\sum_t\lceil N_t/2\rceil$, where $N_t$ is the weight count of matrix $t$; rounding the combined weight count once could miss padding between matrices.

Now use actual learned weights. The default example reuses Chapter 15’s training job for $2$ updates, snapshots that state once, and derives both group sizes from it. There are $22$ learned matrices containing $1544$ weights. Gains, model constants and text assets retain their original payloads. The observed archive counts, all in bytes, are:

Each packed matrix adds a logical shape of two U64 dimensions, or $16$ bytes; $22\times16=352$ accounts for the shape row. The existing SafeTensors container stores code bytes as U8, scales as F64, and logical shapes as U64[2], alongside the native configuration descriptor. These are signed integer codes packed into bytes; the floating-point F4 dtype is a different representation. In this small model, scale and file-header overhead consume much of the code-payload saving. A count of code bytes alone would greatly overstate the reduction in the complete saved file.

Loading reconstructs the ordinary f64 matrices before the existing model uses them. The smaller file therefore does not establish less live inference RAM or faster multiplication. We have measured saved storage and numerical changes, not an integer inference kernel.

Compare the same learned state on fixed documents

The reloaded models use Chapter 16’s fixed comparison documents: aaab and aab retain its train label; abaa and baa retain its validation label. Those labels identify the comparison splits. The Chapter 15 parent job was not trained on these two train strings. Quantization and scoring perform no additional training or tokenizer fitting.

Each split supplies $9$ selected next-token targets in $2$ batches, with zero recorded graphs. The unchanged evaluator measures mean negative log-likelihood (NLL) in nats per selected target. Perplexity is $\mathrm{PPL}=\exp(L)$, where $L$ is that target-weighted mean NLL.

The corresponding train/validation perplexities are $224.927749/232.231355$ for the source, $230.974776/234.052658$ for $g=2$, and $233.175595/235.262392$ for $g=4$. These metrics round to six decimals after calculation. Across the $1544$ learned matrix entries, both quantized forms have maximum absolute error $0.057162531282$; weight MSE is $0.000092861899$ for $g=2$ and $0.000156773622$ for $g=4$.

Both quantized validation NLLs are higher than the source value in this observation: the selected validation targets became less likely on average. The smaller $g=4$ file also has the larger observed weight MSE and validation NLL. This is a tradeoff on four fixed constructed strings, not a population estimate, benchmark result or guarantee about useful language generation. The source settings, named tensors and model identity remain unchanged; each quantized representation has its own derived model identity.

Relate integer representation to inference arithmetic

A shared scale makes the largest magnitude determine the spacing available to smaller values; representation alone also leaves execution costs unresolved.

Jacob and colleagues describe a scaled affine integer representation and the additional arithmetic needed for integer-only inference. The linked author preprint dates from 2017; the work appeared at CVPR 2018. In that author preprint, the affine representation is $r=S(q-Z)$, where $r$ is the represented real value, $q$ is an integer, $S>0$ is a scale and $Z$ is the code corresponding to real zero.

Dettmers and colleagues discuss absolute-maximum quantization and combine vector-wise scaling with mixed-precision treatment of transformer outlier features. Their LLM.int8() paper appeared at NeurIPS 2022. LLM.int8() combines vector-wise scaling with a separate higher-precision path for transformer outlier features. This does not make our tiny matrix an outlier study of a large transformer.

Measure both numerical changes and model scores when reducing stored precision, and distinguish the stored representation from the arithmetic used for inference. Our reference contrast is narrower: keep the original f64 matrix product, then run it again with weights reconstructed from codes and scales.

The row groups, four-bit range, reserved code and tie policy here are course choices. This implementation reproduces neither paper, GPTQ nor AWQ, and makes no claim to their quality or speed results. The Rust contrast below isolates lossy weight storage from execution engineering.

Add the representation around the existing computation

The scalar helper makes the required order visible: validate, divide, round, then clamp. Its supplied scale also lets the fixed-scale clipping example remain separate from group fitting. If division of finite inputs overflows, clamping still saturates to the appropriate endpoint code.

/// Quantize with an already chosen scale. Halfway values round away from zero.
/// A finite input whose ratio overflows still saturates to the appropriate end.
pub fn symmetric_i4_code(value: f64, scale: f64) -> Result<i8, QuantizationError> {
    if !value.is_finite() {
        return Err(QuantizationError::NonFinite);
    }
    if !scale.is_finite() || scale <= 0.0 {
        return Err(QuantizationError::Scale);
    }
    Ok((value / scale).round().clamp(-7.0, 7.0) as i8)
}

GroupedI4Matrix::quantize fits an absolute-maximum scale inside each row group. The single row-major index chooses both the destination byte and its low or high nibble; it continues across row boundaries while scale fitting starts afresh for every row.

    pub fn quantize(matrix: &Tensor, group_size: usize) -> Result<Self, QuantizationError> {
        let shape: [usize; 2] = matrix
            .shape()
            .try_into()
            .map_err(|_| QuantizationError::Shape)?;
        let (count, groups) = layout(shape, group_size)?;
        let mut packed = vec![0; count.div_ceil(2)];
        let mut scales = Vec::with_capacity(groups);
        let mut index = 0;
        for row in matrix.as_slice().chunks_exact(shape[1]) {
            for group in row.chunks(group_size) {
                if group.iter().any(|value| !value.is_finite()) {
                    return Err(QuantizationError::NonFinite);
                }
                let maximum = group.iter().map(|value| value.abs()).fold(0.0, f64::max);
                let scale = if maximum == 0.0 { 1.0 } else { maximum / 7.0 };
                for &value in group {
                    let code = symmetric_i4_code(value, scale)?;
                    packed[index / 2] |= ((code as u8) & 0x0f) << (4 * (index % 2));
                    index += 1;
                }
                scales.push(scale);
            }
        }
        Self::from_parts(shape, group_size, packed, scales)
    }

Reading packed data requires the same invariants as writing it. A wrong code length or scale count would associate weights with the wrong bytes or scales; an invalid scale or overflowing reconstruction would not produce a finite logical matrix. The constructor also rejects a reserved code or nonzero padding instead of silently treating either as a weight.

    pub fn from_parts(
        shape: [usize; 2],
        group_size: usize,
        packed: Vec<u8>,
        scales: Vec<f64>,
    ) -> Result<Self, QuantizationError> {
        let (count, groups) = layout(shape, group_size)?;
        if packed.len() != count.div_ceil(2) {
            return Err(QuantizationError::PayloadLength);
        }
        if scales.len() != groups {
            return Err(QuantizationError::ScaleCount);
        }
        if scales.iter().any(|s| !s.is_finite() || *s <= 0.0) {
            return Err(QuantizationError::Scale);
        }
        if count % 2 == 1 && packed.last().unwrap() & 0xf0 != 0 {
            return Err(QuantizationError::Padding);
        }
        let result = Self {
            shape,
            group_size,
            packed,
            scales,
        };
        for index in 0..count {
            if result.code(index) == -8 {
                return Err(QuantizationError::ReservedCode);
            }
            if !result.value(index).is_finite() {
                return Err(QuantizationError::NonFinite);
            }
        }
        Ok(result)
    }

Dequantization multiplies each signed code by its row group’s scale and returns the existing Tensor. The matrix-product probe then calls the same matmul on the source and reconstructed matrices.

    /// Reconstruct the logical matrix for the existing floating-point operations.
    pub fn dequantize(&self) -> Result<Tensor, QuantizationError> {
        let count = self.shape[0] * self.shape[1];
        Tensor::from_vec(
            self.shape.to_vec(),
            (0..count).map(|i| self.value(i)).collect(),
        )
        .map_err(QuantizationError::Layout)
    }

pub fn matrix_probe(config: &Config, group_size: usize) -> DemoResult<Value> {
    let weights = Tensor::from_vec(
        integers(config, "probe.shape")?,
        floats(config, "probe.weights")?,
    )?;
    let input_values = floats(config, "probe.input")?;
    let input = Tensor::from_vec(vec![1, input_values.len()], input_values)?;
    let encoded = GroupedI4Matrix::quantize(&weights, group_size)?;
    let decoded = encoded.dequantize()?;
    let original_output = matmul(&input.view(), &weights.view())?;
    let decoded_output = matmul(&input.view(), &decoded.view())?;
    Ok(json!({
        "shape":weights.shape(),"group_size":group_size,
        "scales":encoded.scales(),"codes":encoded.codes(),
        "packed_hex":encoded.packed().iter().map(|b|format!("{b:02X}")).collect::<Vec<_>>(),
        "decoded":decoded.as_slice(),"weight_error":errors(weights.as_slice(),decoded.as_slice()),
        "input":input.as_slice(),"original_output":original_output.as_slice(),
        "decoded_output":decoded_output.as_slice(),
        "output_error":errors(original_output.as_slice(),decoded_output.as_slice()),
        "f64_weight_bytes":weights.len()*8,"code_bytes":encoded.packed().len(),
        "scale_bytes":encoded.scales().len()*8,
    }))
}

PortableModel::quantize_weights derives an archive from all and only the rank-two learned matrices. Other learned gains, model constants and text assets keep their original records. The existing native config-lib settings name the versioned scheme and group size; SafeTensors owns the container format. The method refuses an already quantized source rather than silently quantizing its reconstruction again.

    /// Derive a lossy inference archive; leave the source model unchanged.
    pub fn quantize_weights(&self, group_size: usize) -> ArchiveResult<Self> {
        self.validate()?;
        if self.quantization_group_size()?.is_some() {
            return Err("weights are already quantized; select an unquantized source".into());
        }
        if group_size == 0 {
            return Err("quantization group size must be positive".into());
        }
        let mut result = Self {
            settings: config_lib::Config::from_string(&self.configuration()?, Some("conf"))?,
            tensors: self.tensors.clone(),
        };
        for parameter in self.parameters()? {
            let matrix = parameter.tensor().value();
            if matrix.shape().len() != 2 {
                continue;
            }
            let name = parameter.name();
            let quantized = GroupedI4Matrix::quantize(&matrix, group_size)?;
            result.tensors.insert(
                name.into(),
                NamedData {
                    dtype: Dtype::U8,
                    shape: vec![quantized.packed().len()],
                    bytes: quantized.packed().to_vec(),
                },
            );
            result.tensors.insert(
                format!("{SCALES}{name}"),
                NamedData::f64(vec![quantized.scales().len()], quantized.scales()),
            );
            result.tensors.insert(
                format!("{SHAPES}{name}"),
                NamedData::u64(&[
                    u64::try_from(matrix.shape()[0])?,
                    u64::try_from(matrix.shape()[1])?,
                ]),
            );
        }
        result.settings.set("quantization.scheme", I4_SCHEME)?;
        result
            .settings
            .set("quantization.group_size", i64::try_from(group_size)?)?;
        result.validate()?;
        Ok(result)
    }

Restoration checks the descriptor and typed records, reconstructs logical matrices, and passes them through the existing decoder or backbone constructors. The text embedding and output head still share one live table; a generic backbone archive needs no tokenizer. This is lossy inference storage. Exact training resume retains Chapter 15’s unquantized model/optimizer/random-state compatibility envelope and rejects packed model state, even inside an otherwise checksum-valid training checkpoint.

    pub(super) fn logical_parameter(
        &self,
        name: &str,
        data: &NamedData,
        group_size: Option<usize>,
    ) -> ArchiveResult<Tensor> {
        if let Some(group_size) = group_size.filter(|_| name.ends_with(".weight")) {
            let shape = self
                .tensors
                .get(&format!("{SHAPES}{name}"))
                .ok_or("missing quantized logical shape")?;
            if data.dtype != Dtype::U8 || data.shape != [data.bytes.len()] || shape.shape != [2] {
                return Err("invalid packed matrix dtype or shape".into());
            }
            let dimensions = shape.unsigned_values()?;
            if dimensions.len() != 2 {
                return Err("invalid quantized logical shape payload".into());
            }
            let scales = self
                .tensors
                .get(&format!("{SCALES}{name}"))
                .ok_or("missing quantized matrix scales")?;
            let scale_values = scales.float_values()?;
            if scales.shape != [scale_values.len()] {
                return Err("invalid quantization scale shape".into());
            }
            return Ok(GroupedI4Matrix::from_parts(
                [
                    usize::try_from(dimensions[0])?,
                    usize::try_from(dimensions[1])?,
                ],
                group_size,
                data.bytes.clone(),
                scale_values,
            )?
            .dequantize()?);
        }
        Ok(Tensor::from_vec(data.shape.clone(), data.float_values()?)?)
    }

The default comparison reuses Chapter 15’s fresh_job and advance and Chapter 16’s fixture_from_tokenizer and score_model. Each group size is serialized and reloaded before comparison. There is one training algorithm, one evaluator and one generator; quantization changes their input weights.

pub fn demo() -> DemoResult<Value> {
    let config = settings()?;
    let mut job = fresh_job()?;
    let updates = number(&config, "reference_updates")?;
    advance(&mut job, updates)?;
    let source = PortableModel::from_text(&job.model, &job.tokenizer)?;
    let source_id = source.model_id()?;
    let fixture = fixture_from_tokenizer(job.tokenizer.clone())?;
    let mut probes = Vec::new();
    let mut models = Vec::new();
    for group_size in integers(&config, "group_sizes")? {
        probes.push(matrix_probe(&config, group_size)?);
        let encoded = source.quantize_weights(group_size)?;
        let loaded = PortableModel::from_bytes(&encoded.to_bytes()?)?;
        models.push(comparison(&source, &loaded)?);
    }
    Ok(
        json!({"chapter":"17-weight-quantization","reference_updates":updates,
        "comparison_note":"Fixed Chapter 16 comparison documents; train is their split label, not the Chapter 15 parent's training corpus.",
        "source":{"model_id":source_id,"storage":storage_record(&source)?,
            "comparison_documents":score_model(&job.model,&fixture,1)?},
        "matrix_probes":probes,"quantized_models":models,
        "source_unchanged":source.model_id()?==source_id}),
    )
}

The thin clap command accepts a saved model, derives packed weights and saves the new archive. Its comparison uses the saved text model’s restored tokenizer and the same fixed comparison documents; it does not retrain.

pub fn quantize_saved(model: &Path, output: &Path, group_size: usize) -> DemoResult<Value> {
    let source = PortableModel::load(model)?;
    let encoded = source.quantize_weights(group_size)?;
    encoded.save_atomic(output)?;
    Ok(
        json!({"source_model_id":source.model_id()?,"source_storage":storage_record(&source)?,
        "saved":true,"derived":comparison(&source,&encoded)?}),
    )
}

Follow two representations of the same matrix

The figure compares representations against one unchanged matrix and product. The learned-model NLL comparison above is a separate observation of how reconstructed weights affect selected token probabilities.

Reproduce the report and load the saved packed model

From the repository root, with Docker and the existing course Docker setup available, run the default CPU example. The matrix, native configuration and small training job are included; no external corpus or downloaded model is needed.

COURSE_CORPUS=false ./course run cargo run --quiet --offline --locked -p practical-ch17-weight-quantization

The JSON contains matrix_probes, source, quantized_models, reference_updates: 2 and source_unchanged: true. It reports both group sizes, their packed bytes, errors, storage and fixed-document scores. The default serialized-model comparisons reload bytes in the same program.

To use actual saved files across separate programs, keep the same repository-root working directory and Docker setup and run this entire block as one command. One Docker session retains the files while each Rust program exits before the next program starts.

COURSE_CORPUS=false ./course run sh -c '
set -eu
mkdir -p tmp/ch17-quantization-flow
cargo run --quiet --offline --locked -p practical-ch15-save-load-resume-model -- train --until 2 --model tmp/ch17-quantization-flow/parent.safetensors --checkpoint tmp/ch17-quantization-flow/parent.train
cargo run --quiet --offline --locked -p practical-ch17-weight-quantization -- quantize --model tmp/ch17-quantization-flow/parent.safetensors --output tmp/ch17-quantization-flow/group4.safetensors --group-size 4
cargo run --quiet --offline --locked -p practical-ch15-save-load-resume-model -- infer --model tmp/ch17-quantization-flow/group4.safetensors --prompt ab --tokens 3
cargo run --quiet --offline --locked -p practical-ch16-language-model-evaluation -- evaluate --model tmp/ch17-quantization-flow/group4.safetensors --batch-rows 1
'

The first program saves the source model after $2$ updates and its separate training checkpoint. The second saves the $g=4$ inference archive; the unchanged Chapter 15 inference and Chapter 16 evaluation commands then load that archive without training. The quantizer’s model_id, inference’s model_id and evaluator’s model_sha256 all identify the derived model as ba40116d31d1d7c25b63fc8b6dca03d0e4a510d3efdc3a29a50ec058948fa12f.

Inference reports device: "cpu", precision: "f64" and tied_alias: true. Its generated IDs are [108,27,159]; the corresponding bytes [106,25,157] do not form useful prose. Evaluation prints validation NLL $5.460701$ and perplexity $235.262392$, with $9$ selected targets, $2$ batches and zero recorded graphs. Equal printed metrics establish agreement at the stated six-decimal display precision. This bounded CPU run does not establish GPU performance or useful language generation. Separate ./course run invocations would use disposable containers and lose the saved files between programs.

With the same repository-root working directory and Docker setup, run the demo’s library tests:

COURSE_CORPUS=false ./course run cargo test --offline --locked -p practical-ch17-weight-quantization

The library result is two passed tests: the constructed packing/product comparison and the actual trained-model archive/evaluation comparison.

Optional inspection and explanation practice

1. Inspect the default command’s two matrix probes. Explain why the source weight $3.5$ reconstructs as $4$ for $g=4$ and as $3.5$ for $g=2$. Contrast that fitted-scale case with clipping source weight $10$ at fixed scale $1$.
2. Decode byte C9 in low-nibble-first order. For source row [-7,-3.5,0,3.5,7] at $g=4$, explain bytes C9 40 07 and distinguish the final shorter group from the padding nibble.
3. Use the default report’s reconstructed products and error summaries for groups of $2$ and $4$ weights to explain why product MSE differs even though their maximum absolute weight errors are equal.
4. Compare the source model and groups of $2$ and $4$ weights in the default report: complete files, code payloads, scale payloads and validation scores. Explain the storage/error tradeoff and the limit of the fixed-document quality comparison.
5. Run the complete single-session save/quantize/infer/evaluate block and the two-test command above. Check the derived model identity, CPU precision, tied table and validation score reported by the separate consumers.

Keep the model path while changing its saved representation

The learned matrices now have an explicit code-and-scale storage option. The same text encoder, source-independent backbone and tied head load their reconstructed weights and use the existing CPU evaluator and generator. The comparison keeps smaller saved storage distinct from numerical error, model likelihood, live memory and execution speed. Exact training continuation still needs the original unquantized training state.

Planned Chapter 18, Grouped-query attention and context masks, will extend attention sharing and the context permitted in full and cached computation. That changes a different part of the model than the weight representation studied here.