From assigned probability to perplexity

Start with the probabilities of what actually happened

Chapter 6 built a smoothed bigram model. For each current token, it can compute a probability distribution over possible continuations. To score an observed transition, use the probability of the next token that actually occurred, not the maximum probability in that distribution. Suppose two successive targets received these probabilities:

Before introducing the metric, make three predictions:

1. Which of the two targets should count as more surprising?
2. The two probabilities multiply to $1/8$. Why is that product not yet a fair per-target score for comparing sequences with different numbers of targets?
3. If both positions had received one constant probability, what value would represent the same overall difficulty?

For an event assigned probability $p$, define its surprise here as $-\ln p$. The logarithm is natural. The two positions therefore contribute

The second target confirms the first prediction: a smaller assigned probability means greater surprise. Under these fixed contexts, $1/2\times1/4=1/8$ is the likelihood that the model assigns to the two observed targets together. Taking the negative logarithm turns that product into a sum: $-\ln((1/2)\times(1/4))=-\ln(1/2)-\ln(1/4)=\ln 2+\ln 4$. This is why the sum is called negative log-likelihood. Its value, $\ln 8=2.079441541680$, is total surprise for two targets.

Every additional observed target multiplies the sequence likelihood by another probability and adds another surprise term. A longer sequence therefore usually has a smaller raw likelihood even when the model performs equally well per target. To remove that length effect, divide total surprise by the number of observed targets. Here the result is mean negative log-likelihood (mean NLL), $1.039720770840$ nats per target. Exponentiating it gives perplexity $2.828427124746$.

The constant probability requested in the third prediction is $1/\operatorname{PPL}$, approximately $0.353553390593$. If the model assigned that probability at both positions, the geometric mean of the assigned probabilities—and therefore perplexity—would remain unchanged. Equivalently, this perplexity is the effective number of equally likely continuations that would produce the same average surprise. It does not mean that the model literally had $2.828427124746$ choices.

Two simple cases make the scale easier to understand. If every observed target gets probability one, every term is $-\ln 1=0$, so mean NLL is $0$ and perplexity is $1$. If a model is uniform over a vocabulary $V$, every observed target gets $1/|V|$; each surprise is $\ln |V|$, so mean NLL is $\ln |V|$ and perplexity is $|V|$. For $|V|=5$, these formulas give $1.609437912434$ nats per target and perplexity $5.000000000000$.

Before writing the general formula, consider a common averaging mistake. Document A has one target assigned probability $1$. Document B has three targets, each assigned probability $1/4$. Written target by target, the surprises are

The correct mean is $(0+3\ln 4)/4=1.039720770840$, with perplexity $2.828427124746$. If we instead compute a mean for each document and then average those two values, both documents receive equal weight. That gives the incorrect mean $(0+\ln 4)/2=0.693147180560$ and perplexity $2$. The result is wrong because Document B contributes three scored targets while Document A contributes only one. The fact that the correct result matches the earlier two-target perplexity is a coincidence of these particular numbers, not a general rule.

Sum surprise, divide by targets, then exponentiate

The complete rule is:

At target position $t$, $z_t$ is the token that occurred, and $p_t(z_t)$ is the probability assigned to it under the fixed context-construction rule. The metric uses that specific probability—not the largest probability in the distribution and not the entropy of the whole distribution.

In this chapter $\log$ means the natural logarithm, so $\log p=\ln p$. For a valid positive probability no greater than one, $\ln p$ is nonpositive. The leading minus sign turns it into nonnegative surprise: decreasing $p$ increases $-\ln p$. We sum that quantity for every scored target, then divide once by $N$, the total number of target tokens across all scored documents. This is the length normalization missing from a raw sequence product.

The mean $\mathcal{L}$ is measured in nats per target. Perplexity is dimensionless; it is obtained by applying $\exp$ to that mean. When every assigned probability is positive, the same relation can be written

This equation does not define a second measurement. Perplexity is the geometric mean of the inverse assigned probabilities already used in mean NLL. Equivalently, $1/\operatorname{PPL}$ is the geometric mean of the original assigned probabilities.

Mean NLL can also be described as empirical cross-entropy, but only under explicit conditions. For each fixed context, define an empirical distribution that assigns probability $1$ to the token that actually occurred and $0$ to every other token. The cross-entropy between this distribution and the model distribution is exactly $-\log p_t(z_t)$. Averaging those values over the same contexts and the same target tokens gives the mean NLL above. If the empirical distribution and the rule for building context have not been stated, the clearer name is mean NLL on these targets, not simply “cross-entropy.”

A zero probability from the model is different from invalid input. If the model assigns probability $0$ to a token that occurred, it has declared that event impossible. Zero is valid input to the metric, but it makes that token’s surprise, the resulting mean NLL, and perplexity positive infinity. The implementation does not replace zero with an $\epsilon$.

An empty slice, NaN, positive or negative infinity supplied as a probability, or a value outside $[0,1]$ is an input error. Rust validates the whole slice before summing, so [0, NaN] reports the invalid NaN instead of returning infinity because of the earlier zero. Add-one smoothing makes the bigram probabilities used later positive; it does not change this general rule.

Locate every symbol in the worked example

The formula follows mathematical one-based indexing, while Rust slices and the diagram trace are zero-based. Trace index=0 is the first observed target, $t=1$; trace index=1 is the second, $t=2$. Nothing is shifted, and index zero does not denote a boundary token.

Across corpus documents, BOS supplies the first context and is not counted in $N$. Each document’s EOS is its final target and is counted. Because documents remain separate, no fabricated EOS→BOS transition contributes to either the sum or the denominator.

Add logarithms instead of multiplying probabilities directly

Shannon’s 1948 paper develops logarithmic measures in connection with information, choice, and uncertainty. The complete Harvard copy of the paper provides the facts needed here. Its Introduction (viewer pages 1–2) explains that the logarithm base determines the unit. Section 6 (viewer and printed pages 10–11) shows that certainty has zero discrete entropy and that a uniform distribution over $n$ outcomes reaches the maximum $\log n$. This is information-theory background; Shannon does not define language-model mean NLL or perplexity.

The JMLR page for Bengio, Ducharme, Vincent, and Jauvin (2003) links their account of a neural probabilistic language model. In the paper’s PDF, §1.1 (printed page 1139) writes a word-sequence probability as a product of conditional next-word probabilities. Section 2 (printed page 1141) describes perplexity both as the geometric mean of inverse assigned probabilities and as the exponential of average negative log-likelihood. Section 4 (printed pages 1148–1149) states which tokens enter the reported averages before comparing named models on the Brown and AP News corpora. Those results apply only to the named experiments; they establish neither a universal model ranking nor a rule for comparing perplexities across tokenizers.

For this course, we use natural logarithms, give every target token equal weight, define the behavior for $p=0$ and invalid input, keep documents separate, and allow the Chapter 7 bigram scorer to evaluate only training or validation data. We compare perplexities only when the tokenizer, the mapping from token IDs to token values, the document-boundary rules, the rule for building context, and the exact evaluated targets all match. These are course design decisions, not claims drawn from either paper.

The Chapter 7 Rust program shows why logarithms are useful. Multiplying $2{,}000$ factors of $1/2$ in f64 eventually rounds the product to 0.000e0, even though every factor is valid. The sum of the negative logarithms remains finite. This conversion does not change which sequence is more likely: a larger likelihood corresponds to a smaller total surprise. Only the floating-point representation changes—from a fragile product to a stable sum.

A separate example shows why checking only $\arg\max$ loses information. Let the observed target be B:

q = [A: 0.60, B: 0.30, C: 0.10]
r = [A: 0.60, B: 0.20, C: 0.20]

Both distributions rank A highest. However, $q$ assigns probability $0.30$ to the observed token B, while $r$ assigns it $0.20$. Their NLL values for B are therefore $1.203972804326$ and $1.609437912434$. The NLL under $q$ is lower because $q$ assigned more probability to the token that occurred. This example illustrates a limitation of $\arg\max$; it does not claim that perplexity historically replaced accuracy everywhere.

Implement the metric once and reuse it for the frozen bigram

The generic metric function first checks that a nonempty slice contains only finite probabilities in [0,1]. Only after validating the entire slice does it add each selected target probability to the shared accumulator. finish divides total surprise by the target count and exponentiates the resulting mean. An assigned zero contributes positive infinity; an assigned one contributes exactly +0.0.

#[derive(Clone, Copy, Debug, Default)]
struct MetricAccumulator {
    total_surprise: f64,
    target_count: usize,
}

impl MetricAccumulator {
    fn observe(&mut self, probability: f64) {
        self.target_count += 1;
        if probability == 0.0 {
            self.total_surprise = f64::INFINITY;
        } else if probability == 1.0 {
            // Preserve positive zero for exact learner-facing output.
            self.total_surprise += 0.0;
        } else {
            self.total_surprise += -probability.ln();
        }
    }

    fn finish(self) -> Result<MetricSummary, MetricError> {
        if self.target_count == 0 {
            return Err(MetricError::EmptyTargets);
        }

        let mean_nll = self.total_surprise / self.target_count as f64;
        Ok(MetricSummary {
            total_surprise: self.total_surprise,
            target_count: self.target_count,
            mean_nll,
            perplexity: mean_nll.exp(),
        })
    }
}

/// Scores probabilities assigned to observed targets using natural logarithms.
///
/// The complete slice is validated before accumulation. Both `0.0` and `-0.0`
/// are valid impossible-evidence values and produce positive infinity without a
/// clamp.
pub fn score_assigned_probabilities(probabilities: &[f64]) -> Result<MetricSummary, MetricError> {
    if probabilities.is_empty() {
        return Err(MetricError::EmptyTargets);
    }

    for (index, &probability) in probabilities.iter().enumerate() {
        if !probability.is_finite() || !(0.0..=1.0).contains(&probability) {
            return Err(MetricError::InvalidProbability { index, probability });
        }
    }

    let mut accumulator = MetricAccumulator::default();
    for &probability in probabilities {
        accumulator.observe(probability);
    }
    accumulator.finish()
}

The bigram adapter does not implement another formula. It takes a reference to the already fitted BigramModel, walks windows(2) inside each encoded document, queries the probability assigned to each observed next token, and feeds it to the same private accumulator. score_bigram_partition accepts a ScoredPartition; that enum contains only Train and Validation, so the function cannot select the test partition.

/// A partition Chapter 7 is permitted to score.
///
/// The test partition is intentionally unavailable until Chapter 34.
///
/// ```compile_fail,E0599
/// use llm_from_scratch::metrics::ScoredPartition;
///
/// let _missing_variant: ScoredPartition = ScoredPartition::Test;
/// ```
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScoredPartition {
    Train,
    Validation,
}

impl ScoredPartition {
    const fn corpus_partition(self) -> Partition {
        match self {
            Self::Train => Partition::Train,
            Self::Validation => Partition::Validation,
        }
    }
}

/// Scores adjacent target transitions without refitting or joining documents.
pub fn score_bigram_partition(
    model: &BigramModel,
    partitions: &EncodedCorpusPartitions,
    partition: ScoredPartition,
) -> Result<PartitionScore, MetricError> {
    let documents = partitions.documents(partition.corpus_partition());
    let mut accumulator = MetricAccumulator::default();

    for document in documents {
        for transition in document.token_ids().windows(2) {
            let probability = model.smoothed_probability(transition[0], transition[1])?;
            accumulator.observe(probability);
        }
    }

    Ok(PartitionScore {
        partition,
        document_count: documents.len(),
        metrics: accumulator.finish()?,
    })
}

To make the results reproducible, the setup code loads the corpus and split definition stored in the repository, learns eight BPE merge ranks from the training documents, creates tokenizer layout 1 with a vocabulary of 266 tokens, and fits one bigram model with $\alpha=1$ from eight training documents and 1,844 transitions. The excerpt below shows that sequence. Checks later in the same file stop with an error if the corpus checksum, document IDs, tokenizer settings, or model-training setup differ from the expected values.

/// Executes the data-to-model path before the frozen identities are checked.
fn reconstruct_frozen_metric_fixture() -> Result<ReconstructedFixture, FrozenFixtureError> {
    let corpus = Corpus::from_json(CORPUS_JSON)?;
    let manifest = SplitManifest::from_json(SPLIT_MANIFEST_SOURCE)?;
    let source_partitions = manifest.partition(&corpus)?;
    let training = BpeTrainer::new(EXPECTED_REQUESTED_MERGES).train(&source_partitions)?;
    let tokenizer = BpeTokenizer::from_training(&training)?;
    let encoded_partitions =
        EncodedCorpusPartitions::from_partitions(&source_partitions, &tokenizer);
    let model = BigramModel::fit_encoded_training_partition(
        tokenizer.layout().vocabulary_size(),
        FIT_ALPHA,
        &encoded_partitions,
    )?;

    Ok(ReconstructedFixture {
        corpus,
        manifest,
        training,
        tokenizer,
        encoded_partitions,
        model,
    })
}

The following main.rs excerpt first computes metrics for the $[1/2,1/4]$ example and for the perfect and uniform scale anchors. It also covers zero probability and empty input. Next, it contrasts correct and incorrect document weighting, compares two distributions with the same argmax, and demonstrates why a direct probability product can underflow. Finally, it scores the same unchanged model separately on training and validation. Code outside the displayed region prints the returned results. While printing the result for $2{,}000$ factors of $1/2$, it also checks whether the already-computed total surprise is finite. That check is not a new metric calculation: no probability sequence or corpus partition is scored again.

    let tiny_probabilities = [0.5, 0.25];
    let tiny = score_assigned_probabilities(&tiny_probabilities)?;
    let perfect = score_assigned_probabilities(&[1.0, 1.0])?;
    let uniform = score_assigned_probabilities(&[1.0 / 5.0; 5])?;
    let impossible = score_assigned_probabilities(&[0.8, 0.0])?;
    let empty_error = match score_assigned_probabilities(&[]) {
        Err(error) => error,
        Ok(_) => return Err("empty metric input unexpectedly produced a score".into()),
    };

    let weighted_documents = score_assigned_probabilities(&[1.0, 0.25, 0.25, 0.25])?;
    // Deliberate misuse: because each document has one constant assigned
    // probability, [1.0, 0.25] represents their two geometric means. Scoring
    // that two-item pseudo-sample gives documents equal weight instead of
    // weighting all four observed targets.
    let wrong_equal_document_means = score_assigned_probabilities(&[1.0, 0.25])?;

    let q_distribution = [0.60, 0.30, 0.10];
    let r_distribution = [0.60, 0.20, 0.20];
    let q_target_b = score_assigned_probabilities(&[q_distribution[1]])?;
    let r_target_b = score_assigned_probabilities(&[r_distribution[1]])?;
    let lower = if q_target_b.mean_nll() < r_target_b.mean_nll() {
        "q"
    } else {
        "r"
    };

    let halves = vec![0.5; 2_000];
    let raw_product = halves.iter().copied().product::<f64>();
    let halves_score = score_assigned_probabilities(&halves)?;

    let fixture = frozen_metric_fixture()?;
    let train = score_bigram_partition(
        fixture.model(),
        fixture.encoded_partitions(),
        ScoredPartition::Train,
    )?;
    let validation = score_bigram_partition(
        fixture.model(),
        fixture.encoded_partitions(),
        ScoredPartition::Validation,
    )?;

The separate executable at examples/diagram_trace.rs calls render_language_model_metrics_trace in src/diagram_trace.rs and prints the returned string. The displayed region below calls the same metric and fixture functions and computes the two partition scores. It also checks three properties of the training and validation token sequences: BOS never appears as a target, EOS occupies the final target position of every document, and no EOS→BOS pair is introduced. Code after the displayed region serializes those results as ten trace records. It adds test_selectable=no as a literal during serialization; the restriction itself is enforced by ScoredPartition, which has no Test variant. All ten trace records therefore preserve evidence produced by those metric, fixture, and boundary checks; no second metric is calculated for the figure.

    let tiny_probabilities = [0.5, 0.25];
    let first_target = score_assigned_probabilities(&tiny_probabilities[0..1])?;
    let second_target = score_assigned_probabilities(&tiny_probabilities[1..2])?;
    let tiny = score_assigned_probabilities(&tiny_probabilities)?;

    let fixture = frozen_metric_fixture()?;
    let train = score_bigram_partition(
        fixture.model(),
        fixture.encoded_partitions(),
        ScoredPartition::Train,
    )?;
    let validation = score_bigram_partition(
        fixture.model(),
        fixture.encoded_partitions(),
        ScoredPartition::Validation,
    )?;
    let (bos_is_target, eos_is_target, has_cross_document_pair) =
        boundary_evidence(fixture.encoded_partitions());

Run the Chapter 7 program:

./course run cargo run --quiet --locked -p ch07-language-model-metrics

For the same unchanged model, the output reports training mean NLL 3.832941183107 and perplexity 46.198216022322 across 1,844 targets, plus validation mean NLL 3.981939680567 and perplexity 53.620940919077 across 469 targets. These values are deterministic for the recorded corpus, split, tokenizer, model, and evaluation rules. Because training and validation contain different target sets, the gap applies only to these data and does not by itself support a general conclusion about model quality elsewhere. Scoring validation neither updates nor refits the model.

Trace each displayed value back to its Rust calculation

As you read the figure, answer five questions about how its values are related:

1. Which trace row corresponds to mathematical target $t=1$, and where is its assigned probability converted to surprise?
2. Where does target_count=2 enter the calculation, and why is it displayed separately from total surprise?
3. Which stage converts mean NLL to perplexity without requesting additional probabilities from the model?
4. Were the training and validation rows produced by one fitted model or two?
5. Which boundary record shows that score_bigram_partition cannot select the test partition?

The first panel reproduces the opening $[1/2,1/4]$ example from the Rust trace. Rows index=0 and index=1 contain the two probabilities and surprises, while the aggregate shows both total surprise and its two-target denominator. The final stage applies $\exp$ to the computed mean NLL. It requests no new model probabilities; perplexity and mean NLL determine each other uniquely.

The second panel applies the same metric to training and validation. A shared setup block shows that the model was fitted on the training documents and then kept unchanged for both scores. The boundary list shows exactly which tokens are counted: BOS supplies context only, EOS is a target, documents remain separate, and the Chapter 7 bigram scorer cannot select test data.

Predict each result before checking the reasoning

Work on paper first. Show the denominator in averaging problems and the document boundaries in sequence problems.

1. For assigned probabilities $[1/2,1/4]$, compute both surprises, total surprise, target count, mean NLL, and perplexity.
2. Derive the mean NLL and perplexity for perfect assignments and for a uniform distribution over a vocabulary of size $|V|$.
3. Predict the result for $[0.8,0]$. Why would replacing zero with an $\epsilon$ change the metric rather than merely stabilize it?
4. One document has one target at $p=1$; another has three targets at $p=1/4$. Repair an implementation that averages their two document means equally.
5. In [BOS,A,B,EOS], identify every scored target and explain the role of BOS.
6. Join [BOS,A,EOS] and [BOS,B,EOS]. Which artificial transition is created at the join?
7. For observed target B, compare $q=[0.60,0.30,0.10]$ and $r=[0.60,0.20,0.20]$. Why can their NLLs differ even though both have $\arg\max=A$?
8. Can you directly compare perplexities produced with different tokenizers or different evaluated target sets? State every evaluation condition that must match.
9. Why are the training and validation scores of the bigram model with $\alpha=1$ finite even though the general metric maps an assigned zero to positive infinity?

Misconception check: perplexity and mean NLL are not independent measurements. Perplexity is exactly $\exp(\mathcal{L})$, so either value determines the other. The value $1/\operatorname{PPL}$ is the constant probability that would produce the same mean NLL if assigned to every target. Computing perplexity does not request any additional probabilities from the model.

Keep one measurement while the model becomes more capable

Chapter 7 can now report mean NLL and perplexity for the unchanged Chapter 6 model on the exact training and validation targets. Both reports use the same metric and model: scoring neither updates the transition counts nor refits the table, and the Chapter 7 bigram-scoring API offers no test-partition choice.

Chapter 8 begins the numerical engine with tensor values stored in a flat Vec<f64>, plus shapes, strides, and checked coordinate-to-offset indexing. Chapters 8–22 build the tensor, differentiation, and optimization machinery that later models need in order to improve mean NLL.

Chapter 33 will train the complete decoder and select one state using validation loss only. Chapter 34 will freeze that state and demonstrate one local post-selection evaluation on a fixed teaching fixture, comparing it with the retained bigram model. That describes the order inside the demonstrated execution; it makes no claim about how often the fixed result has been used during repository development. A fair comparison requires both models to use the same tokenizer, the same mapping from token IDs to token values, the same exact target set, the same document-boundary and context-construction rules, and the same source data. Chapter 7 supplies the measurement; the later chapters determine how to improve a model and conduct a fair final comparison.