Lowering a language model's loss means compressing the text better

A probability distribution and a code are the same object seen from two sides. Give each outcome about −log⁡2q(x)-\log_2 q(x) bits (rounded up to whole bits, the lengths still fit a prefix-free code) and qq becomes a code. Read any prefix-free code's lengths back as q(x)=2−ℓ(x)q(x) = 2^{-\ell(x)} and the code becomes the distribution it is betting on [1]. So predicting well and compressing well are one skill.

Betting wrong has an exact price. Coding data from the true distribution pp with the code for qq costs, on average, the cross-entropy:

Ep[−log⁡2q(X)]=H(p)+DKL(p ∥ q).\mathbb{E}_p[-\log_2 q(X)] = H(p) + D_{\mathrm{KL}}(p \,\|\, q).

Entropy is the floor: by Shannon's source coding theorem no code averages fewer bits, and the code built from pp itself, the optimal code, reaches it to within a bit per symbol. The KL is the surcharge for the wrong bet, zero only when q=pq = p. Cover and Thomas's four-symbol example makes it concrete. For probabilities 1/2, 1/4, 1/8 and 1/8, the code 0, 10, 110, 111 averages 1.75 bits, exactly the entropy. Swap the first and last symbols' probabilities and the same code averages 2.5 bits: 1.75 of entropy plus 0.75 of KL.

Cross-entropy as entropy plus KLFour bars, one per symbol, with width equal to its probability and height equal to its code length, so total area is the average code length. The data's probabilities are one eighth, one quarter, one eighth and one half; the code 0, 10, 110, 111 was built for the reverse. Dashed outlines show the lengths of the code built for the data, whose area is the 1.75-bit entropy. The code used spends 2 extra bits on the likeliest symbol, adding 1.00 bit on average, and saves 2 bits on the rarest, removing 0.25. The net 0.75 bits is the KL divergence; the total is 2.5 bits.2.5 bits = 1.75 entropy + 0.75 KLwidth = probability, height = code bits, area = average bits0123−0.25010110+1.00111
  • right code: area 1.75 = entropy
  • extra bits: +1.00
  • bits saved: −0.25
The code 0, 10, 110, 111 on data whose likeliest symbol is the last. The KL is the extra area minus the saved area: 0.75 bits.

A language model makes this bet at every token, and arithmetic coding turns its predictions into a real code: a whole text encodes in less than 2 bits more than the model's summed loss in bits [2]. So a language model's loss on any text, a training batch or a held-out document, is the size that text compresses to with the model. Lowering the loss makes that file smaller, so training is compression. And since the loss is entropy plus KL, and the entropy belongs to the text's true distribution, not the model, every bit training saves comes out of the KL: the model's predictions moving closer to the truth.

That file size assumes the receiver already holds the model. A model fitted to the very text it compresses has to be sent too, and pricing that is where minimum description length starts.

References

  1. Elements of Information Theory
    Cover, T. M. and Thomas, J. A., 2006. Wiley-Interscience.

  2. Language Modeling Is Compression [PDF]
    Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M. and Veness, J., 2023. arXiv. DOI: 10.48550/ARXIV.2309.10668