Planning can't fix a wrong world model; only learning from experience can

Reinforcement learning splits how an agent gets better into two kinds [1]. The agent's model of its environment predicts, for a state and an action, the next state and reward. The agent acts by its policy, its rule for choosing an action in each state. Planning is computation that uses the model to improve the policy, with no contact with the environment. Learning improves the agent from real experience of the environment itself.

Planning and learningThree boxes: policy at the top, experience at the bottom left, model at the bottom right. Acting turns the policy into experience. Two arrows leave experience: direct RL updates the policy, and model learning updates the model. Planning, drawn dashed, runs from the model to the policy and is the only arrow that does not start from experience.actingdirect RLmodel learningplanningpolicyexperiencemodel
How an agent improves, after Sutton and Barto's Figure 8.1. Learning starts from experience of the real environment and can update the policy straight from it (direct RL), improve the model (model learning), or both; planning, dashed, starts from the model, so it can only work out what the model already says.

With a perfect model, planning is enough. The rules of chess predict every position exactly, so an agent that knows them could in principle play perfectly without seeing a real game. AlphaZero reached superhuman play this way: everything it trained on came from games against itself, played and scored by the rules [2].

With a wrong model, planning can't help. A robot whose simulator gets friction slightly wrong can plan forever and never notice, because every simulated rollout comes from the simulator and inherits its error. Planning works out what the model implies, mistakes included. Only real experience shows where the model is wrong and corrects it. Information theory states the same limit as the data-processing inequality: processing the model's output can't add information about the world beyond what the model already holds [3].

A reasoning language model's chain of thought is planning against the model it has, its own weights. A longer chain of thought works out more of what the weights imply, wrong beliefs included; fixing a wrong belief takes real experience.

References

  1. Reinforcement Learning: An Introduction [HTML]
    Sutton, R. and Barto, A., 2018. MIT Press.

  2. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm [PDF]
    Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K. and Hassabis, D., 2017. arXiv. DOI: 10.48550/ARXIV.1712.01815

  3. Elements of Information Theory
    Cover, T. M. and Thomas, J. A., 2006. Wiley-Interscience.