Memorizing is the right compression of a fact stated only one way

Train a language model on one biography per person and it can recite every bio, yet it can't answer questions about the people. Allen-Zhu and Li found this in a controlled experiment: 9.7% accuracy on held-out people. Give each person five bios, reworded and shuffled, and accuracy rises to 96.6% [1].

The facts weren't missing. They were filed in the wrong place. With one wording, the model recalls each fact from the text just before it. In the paper's example, it learns "the person born on October 2, 1996 works in Menlo Park" rather than that Anya does. A question supplies only the name, so there's nothing to recall from. With varied wordings, the only thing every version shares is the name, and probes show the facts move onto it.

This is the compression view of learning that Ilya Sutskever argues for [2]; its working form for choosing a model, minimum description length, gives the accounting [3]. The best model of the data is the one that makes the model plus the data it leaves unexplained shortest, so a pattern earns a place in the model only if it saves more than it costs. With one bio per person, each fact appears once, after one particular stretch of text. Stored as "after this text comes Menlo Park," it takes one association; stored under Anya's name, it also takes one. Nothing favors the name, and the model uses the text just before the fact. That is as short as any code for this data: memorizing is the right compression. With five bios, the fact follows five different stretches of text but always the same name. Keyed to the text, it takes five associations; keyed to the name, it takes one. Now the name wins, and a question that supplies only the name can find the fact.

References

  1. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction [PDF]
    Allen-Zhu, Z. and Li, Y., 2023. arXiv. DOI: 10.48550/ARXIV.2309.14316

  2. An Observation on Generalization [link]
    Sutskever, I., 2023.

  3. A tutorial introduction to the minimum description length principle [link]
    Grunwald, P., 2004. arXiv. DOI: 10.48550/ARXIV.MATH/0406077