RLHF trains a model against a reward model learned from human preference judgments. That reward has two limits. It scores what the raters prefer, so it inherits their knowledge and biases: a strategy better than the raters' own looks wrong to them. And it is only a proxy for what they prefer. Gao, Schulman and Hilton found that as optimization against it pushes further, its score keeps rising while the true score rises, peaks and falls [1].
Silver and Sutton take aim at the first limit [2]. Relying on human prejudgment, they write, puts "an impenetrable ceiling" on the agent. Their alternative is grounded rewards, signals measured from the environment rather than judged by people. A health agent, for instance, could be rewarded on the user's resting heart rate, sleep and activity. A measurement credits whatever works, so the agent can find strategies no rater would have scored.
Grounding leaves the second limit, and it becomes the question of which reward. The "reward is enough" hypothesis of Silver, Singh, Precup and Sutton holds that maximizing a reward in a rich environment is enough to produce intelligence [3]. They ask which reward, and answer that almost any will do: an agent paid for each round pebble it collects might end up building "a corporation that collects pebbles." That settles whether the agent becomes capable, not whether it does what you want. What is the reward for the health agent? Resting heart rate, sleep and activity are proxies for health, and optimizing a proxy hard exploits the gap between them: Goodhart's law, which reinforcement learning calls specification gaming. OpenAI's CoastRunners agent, rewarded on the boat race's own score, learned to circle a lagoon hitting the same three targets instead of finishing the race [4].
Silver and Sutton's answer is to learn the reward, in what they call bi-level optimization: a function combines the grounded signals into one reward, and the user's reports of how things turned out, such as their satisfaction, tune which signals count and by how much [2]. That isn't the prejudgment they rejected, since the user reports a consequence rather than judging an action in advance. But satisfaction is a proxy too, so the question of which reward moves up a level instead of going away; they grant "no guarantee of perfect alignment."
So reward is enough only once you know the reward. That is settled only where the reward is the goal itself: a game won, a proof checked. For a goal like health, choosing the reward is the unsolved part.
References
Scaling Laws for Reward Model Overoptimization [PDF]
Gao, L., Schulman, J. and Hilton, J., 2022. arXiv. DOI: 10.48550/ARXIV.2210.10760Welcome to the Era of Experience [PDF]
Silver, D. and Sutton, R. S., 2025.Reward is enough [link]
Silver, D., Singh, S., Precup, D. and Sutton, R. S., 2021. Artificial Intelligence, Vol 299, pp. 103535. Elsevier BV. DOI: 10.1016/j.artint.2021.103535Faulty Reward Functions in the Wild [link]
Clark, J. and Amodei, D., 2016. OpenAI.