Entropy & Cross-Entropy

Language models can sound confident about nonsense. Let’s understand why, how we measure what it actually can predict, and what we can do about it.

Language models make predictions under uncertainty.

Entropy measures how much uncertainty there is in what should come next. It’s about the inherent randomness in a situation (regardless of whether anyone is trying to predict it).

Cross-entropy measures how well your model’s predictions match reality given this uncertainty. (Think: “prediction quality score” - lower is better. High cross-entropy = your model’s predictions are further from reality.)

Example: who broke the vase?

You arrive home and find a broken vase. You have two cats (Red and Black), and they are equally mischievous: either could have broken it.

Entropy depends on the situation: there’s a 50%-50% uncertainty on which cat broke the vase. If you had four cats instead of two the entropy would be higher: it’d be even harder to guess correctly which cat broke the vase.

Cross-entropy measures how well your predictions match reality. You know Black holds a grudge, so Black will be more likely to be the culprit. If you take this into account in your prediction, your cross-entropy will be lower (you have a better chance at guessing it right.) If you ignore this information and guess 50-50, your cross-entropy is higher.

Cross-entropy combines the natural unpredictability (entropy) plus how wrong your guesses are. Even with perfect predictions, you can never do better than the natural entropy of the situation.

Understanding entropy and cross-entropy helps you make better decisions about your AI systems.

Some prediction tasks are naturally harder than others. Binary classification (spam / not spam) has lower entropy than multi-class problems (categorizing customer support tickets into 20 types).

Higher entropy tasks need more sophisticated models and have fundamentally lower accuracy ceilings.

High entropy data requires:

• More training data

• More complex models

• More computation time

• More careful evaluation

Cross-entropy is the most common way to measure how well AI models are learning. When you see “loss decreasing” in training reports, that’s usually cross-eentropy going down. Lower cross-entropy means better predictions.

Compare your cross-entropy to the task’s natural entropy:

• Cross-entropy much higher than entropy = your model needs work

• Cross-entropy close to entropy = you’re hitting fundamental limits

• Cross-entropy decreasing but still high = keep training, you’re improving

• Cross-entropy not decreasing = try a different approach

Most ML frameworks (for example: PyTorch) have tools to calculate entropy and cross-entropy

What’s this:

• H(X) is the entropy of the random variable X

• p(xᵢ) is the probability of outcome xᵢ

• n is the number of possible outcomes

• The logarithm base determines the units (base 2 = bits)

What this formula means: For each possible outcome, multiply its probability by the log₂ of its probability, then sum everything up (with a minus sign to make it positive).

Why this log₂ multiplication?

The log₂ calculates how “surprising” each outcome is:

• If p(xᵢ) = 1.0 (certain) → log₂(1.0) = 0 → No surprise

• If p(xᵢ) = 0.5 (coin flip) → log₂(0.5) = -1 → 1 bit of surprise

• If p(xᵢ) = 0.25 (rare) → log₂(0.25) = -2 → 2 bits of surprise

We multiply the probability by its logarithm in order to weigh the surprise with how often it occurs.

• A very surprising event (high -log₂) that almost never happens will have almost no impact on the entropy (because it’s so rare).

• An event that is not surprising at all (because it happens all the time) will have no impact on entropy.

• A moderately surprising event that happens often will impact entropy.

Example: Classifying customer support tickets

In a customer support system where 50% of tickets are billing questions and 50% are technical questions: entropy is 1 (a ticket can be of either type with equal probability)

\(H = -(0.5 \times \log_2(0.5) + 0.5 \times \log_2(0.5)) = 1\)

If 75% are billing and 25% are technical: entropy is lower (a given ticket is likely to be of type billing)

\(H = -(0.75 \times \log_2(0.75) + 0.25 \times \log_2(0.25)) = 0.81\)

What this means: When outcomes are equally likely, entropy is highest (maximum uncertainty). When one outcome dominates, entropy is lower (more predictable). Perfect certainty (100%/0%) gives entropy = 0.

Entropy is literally the expected surprise: the average amount of information you get from each outcome, weighted by how often you see it.

What’s this:

• H(P, Q) is the cross-entropy between the true distribution of reality (P) and the distribution of the model (Q).

• H(P) is the entropy of the true distribution. This is the baseline difficulty.

• D_KL(P||Q) is: how much your model diverges from the truth (see Kullback–Leibler divergence to dig deeper into this.)

  • D_KL(P||Q) = 0 → Your model perfectly matches the true distribution

  • D_KL(P||Q) > 0 → Your model is systematically wrong in some way

  • Higher D_KL(P||Q) → More systematic bias/error

What this formula means:

Cross-entropy = natural difficulty + how much your model adds to that difficulty.

You can never do better than H(P), but poor models make the problem harder by adding a large D_KL on top.

Now you can distinguish between “my model needs work” and “this task is just fundamentally hard”, and make the correct decisions for your product.

The next time you see “loss: 2.3” in your training logs, you’ll know whether you should be happy because things are getting better, or you should go back to the drawing board and try a different approach.