Once models are smarter than we are, humans become the weak supervisor: we label a model's behavior as good or bad, and some of our labels are wrong. A student trained on its teacher's labels should at best reproduce the teacher, mistakes included. If that held, supervising a superhuman model would be a dead end.
Burns and colleagues at OpenAI tested this with a stand-in that can be run today: GPT-4 finetuned on labels from a GPT-2-level model [1]. On NLP benchmarks the student typically closed about half the gap between its teacher and GPT-4 trained on the true labels.
The paper's explanation is that the labels elicit rather than teach. A student fitting the labels can learn one of two functions: the task the labels are aiming at, or the weak model's particular pattern of errors. Pretraining already gave GPT-4 a good representation of the task, so the task is salient, the cheap function to fit. Networks learn the pattern in noisy labels before they memorize the noise [2]; pretraining decides what counts as the pattern. GPT-4 may have no ready representation of where a small model goes wrong: larger students agreed less with the teacher's errors, not more.
The same explanation marks the limit. Elicitation works when the student already represents the task and doesn't represent the teacher's errors. Reward modeling, the step in RLHF that learns which of two responses people prefer, fails the first condition: there the student closed only about 10% of the gap. Making the task salient first, by continuing plain next-token training on the preference data's text without its labels, raised that. A superhuman student may fail the second condition. Trained on human text, it will represent human mistakes well, which could make copying them the cheap fit; the authors name this imitation saliency as a limit of their analogy.
References
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision [PDF]
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I. and Wu, J., 2023. arXiv. DOI: 10.48550/ARXIV.2312.09390A Closer Look at Memorization in Deep Networks [PDF]
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y. and Lacoste-Julien, S., 2017. Proceedings of the 34th International Conference on Machine Learning, Vol 70, pp. 233–242. PMLR.