Turning a Model Into One Number

In one line

Cross entropy is the average of the negated logarithm of the probability the model gave to the correct token, and exponentiating it back gives perplexity. Perplexity is read as "on average, how many ways is it torn between at each position".

Why this was needed

Both training and evaluation ultimately move by looking at one number. If the loss goes down, you keep running, and if it does not, you change something. But in the process of producing that number there are several spots where it goes silently wrong without raising an error.

What you actually see looks like this. The loss at some moment becomes inf and from the next batch nan spreads. Or the loss curve goes down nicely, but the generated text is terrible. Or the perplexity you measured yesterday differs from today's although the model is the same. Or someone else's paper number and yours differ by a factor of two, and you cannot tell which is wrong.

The cause is usually one of four. When you took the logarithm, whether you shifted the labels by one position, whether you removed the padding positions, what the base of the logarithm is. All four are one line of code, and an exception is not raised even when they are wrong. So you cannot see them until you measure directly.

When to take the logarithm

The softmax turns a row of scores into a distribution that sums to 1. What we need is not that probability but the log probability. Then you might think "can't we just compute the probability and then take the log?", but that order collapses at the bottom.

The smallest positive number a double-precision float can hold is around 5e-324. If the correct token's score is far below the rest, its probability drops below that and collapses to 0.0. The logarithm of 0 does not exist. Python's math.log throws an exception there, and if you avoid the exception by putting in -inf, the loss at that position becomes +inf, and the moment you average, the whole sentence is stained with inf.

The answer is not to build the probability at all.

# log p_i = x_i - (max + log sum exp(x - max))
top = max(xs)
lse = top + math.log(sum(math.exp(x - top) for x in xs))
logp = [x - lse for x in xs]

There is no division here. You subtract the large value before exp, so it does not overflow, and you get the log probability by subtraction, so it does not bottom out. However small the probability is, its logarithm is just a small negative number. This is why frameworks sell the softmax and the loss bundled into one function — if you call the two operations separately, information disappears in between.

The softmax stabilization covered in the earlier lab was about blocking the top (exp overflowing). What is blocked here is the bottom. The same subtraction of max is doing two jobs, but the place where it collapses and the symptom differ.

From one position to a sentence

The loss at one position is one line.

loss(t) = -log p(correct token t)

If you gave the correct one probability 1, it is 0, and it grows as the probability shrinks. What you gave to the other tokens is not counted separately. Because the sum is 1, the correct one's share is the rest's share.

The loss of a sentence is the average of those values. The reason it is an average and not a sum is to compare sentences of different lengths. If you measure by the sum, a long sentence is always a bad sentence. It is the same job as the mean in statistics, but what you put in the denominator becomes a problem later.

Perplexity is just a change of scale

A loss value of 1.06 gives no feel. If you exponentiate it back, it becomes a number you can read.

perplexity = exp(mean loss)

There is one way to set the scale. Put in a model that knows nothing. If you give the same score to the whole vocabulary, the probability is 1/V and the loss is log V, so the perplexity is exactly V, the vocabulary size. So if the perplexity is near the vocabulary size, that model has learned nothing, and if it is larger, it is worse than a uniform distribution.

An immediate conclusion comes from here. You cannot compare the perplexities of two models with different vocabularies. That is because the starting point of the scale differs. If the tokenizers differ, the number of pieces into which the same text is split also differs, so even the denominator differs. That is why, before putting a paper's number next to yours, you should first check whether the vocabulary and tokenizer are the same.

The labels are shifted by one position

A language model uses the output at position t to predict the token at position t+1. That is what the decoder of Attention Is All You Need does, and the next-token prediction explained by Hugging Face's generation documentation is the same story.

So when measuring the loss, the logits drop one from the back and the tokens drop one from the front. That is because the last row has no next token to predict, and the first token has no row that predicted it.

What happens if you do not shift by one? No error occurs. It is as if you told the model to "predict the token you are looking at right now", so only the loss comes out bad. Conversely, if you shift twice where you should shift once or shift in the opposite direction, the number can strangely come out good — because you are handing it what it has already seen as the answer. If the loss curve is plausible but the generated output is terrible, look here first.

Padding is a free score

To batch them, you fill short rows to match lengths. Those filler tokens are not content but place markers.

The problem is that filler tokens are too easy to predict. After a sentence ends, the same thing always follows, so the model becomes confident quickly. If you put those positions into the average, the loss goes down, and the more padding a batch has, the lower it goes. The model stays the same, yet just by changing how the batch is composed, the number gets better.

There are two places to remove them. You remove them on the adding side and also on the dividing side. If you miss the dividing side, you divide the loss of the remaining positions by the total length and the value silently gets smaller. This mistake is especially hard to find — because the direction is always toward "getting better", there is no reason to suspect it.

Measuring in bits

If you change the base of the logarithm to 2, the unit becomes bits per token. It is one division.

bits/token = natural-log loss / ln 2

It is only a change of base, so 2 ** 비트 (the Korean word means bits) is the same value as exp(자연로그 손실) (the Korean words mean natural-log loss). You just read the same thing with a different ruler, but the compression literature often writes in bits and deep learning often writes in natural log, so looking only at the numbers it can seem to differ by nearly a factor of two. Before copying down someone else's table, you must check the base.

What it looks like in the field

First, the loss suddenly becomes inf or nan. It is the moment code that builds the probability first and then takes the log meets a very low probability. It goes away if you fix it to get the log probability by subtraction.

Second, the loss goes down but the output is bad. First check whether the label shift is off. If you show the answer in advance and tell it to predict, the loss can go down as much as you like.

Third, the perplexity of the same model differs from run to run. It is very likely that the padding ratio of the evaluation batches changed. If you do the masking properly, the value does not wobble even when the batch composition changes.

Fourth, it differs from someone else's number by a factor of two. Match in turn the base of the logarithm, the vocabulary size, the tokenizer, and whether the denominator is counted in tokens or in words. It is usually one of those.

Fifth, you deployed looking at the loss alone and an incident occurred. Perplexity is "how well it predicts the next token", not "whether it gives useful answers". It is good for confirming that it is going down, but you cannot call it a good model on that alone.

What really matters in practice

• Go straight from logits to log probabilities. The moment you go through probabilities, information disappears at the bottom.
• Set the scale first. Knowing just the one fact that the perplexity of a uniform distribution is the vocabulary size lets you judge whether your own number makes sense.
• Do the masking in both the numerator and the denominator. If you remove it on only one side, the value silently gets better.
• To compare, write down the conditions. Vocabulary size, tokenizer, base of the logarithm and padding handling — if these four are not the same, the two numbers measured different things.

What you will do in the next lab

You grow /root/work/tf-loss/loss.py one step at a time. You do not call a real model; you build the same computation yourself with only the standard library — the system Python of this Pod has no numpy, torch or transformers, and numpy exists only inside /opt/onnx-lab/bin/python. So every number that comes out here was measured with data you built.

You start with a stable log-softmax. Then you build separately a version with deliberately the wrong order, and see with your own eyes that -inf actually appears when you compute the probability first. Only with the two functions side by side can you see where they part ways.

From there you move up to the loss at one position, the average over many positions, and perplexity exponentiated back. You also confirm directly that the perplexity of a model that knows nothing equals the vocabulary size.

The last three steps are the point of this lab. With the same data, you place side by side the numbers with and without shifting the labels and with and without removing the padding. All four numbers come out without error and all four look plausible, yet they differ from each other — that is everything this module wants to show. Finally you convert to a base-2 logarithm and read it in bits per token as well. The grader actually imports your module, pokes at the functions directly with different logits every time, and checks them against values it computes separately.