Model collapse is the loss of the tails, and keeping the real data bounds it

Train a language model, generate a corpus from it, train the next model on that corpus, and repeat. Shumailov et al. did this, and by the ninth generation a prompt about church towers produced a list of jackrabbits with different-colored tails [1]. The gibberish is the end state. What goes first is the tails, the rare events of the original distribution.

The cause is finite sampling. Each generation learns from nn samples of the last one's output, and an event rarer than 1/n1/n is missing from all of them with probability at least about 37%. A model fit to the sample gives a missing event little or no probability, and what gets none can't be sampled back, so each generation can only lose tail, never regain it. The distribution narrows; in the paper's simplest case it ends on a single value.

That loop threw each generation's data away. Gerstgrasser et al. ran it both ways [2]. When each generation's synthetic data replaces the last, each generation adds a fresh estimation error of the same size, so after kk generations test error is about kk times the first model's. When it accumulates alongside the original real data, error stays below about 1.64 times the first model's however many generations run. They prove this for linear regression and see the same split in language models. Dohmatob et al. show why in terms of tails: mixing even a small fraction of real data into tail-cut synthetic data restores the original scaling, because the real data still carries the rare events [3]. In the accumulating loop the real data never leaves the pool, so every rare event it contains is in every generation's training set and can't fall to zero.

References

  1. AI models collapse when trained on recursively generated data [link]
    Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. and Gal, Y., 2024. Nature, Vol 631(8022), pp. 755–759. Springer Science and Business Media LLC. DOI: 10.1038/s41586-024-07566-y

  2. Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data [PDF]
    Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., Roberts, D. A., Yang, D., Donoho, D. L. and Koyejo, S., 2024. arXiv. DOI: 10.48550/ARXIV.2404.01413

  3. A Tale of Tails: Model Collapse as a Change of Scaling Laws [PDF]
    Dohmatob, E., Feng, Y., Yang, P., Charton, F. and Kempe, J., 2024. Proceedings of the 41st International Conference on Machine Learning.