Pretraining computes a loss at every token, so each token can teach the model up to bits, where is the vocabulary size. Reinforcement learning on a checkable task samples a rollout that can run to hundreds of thousands of tokens and then scores it once: pass or fail. A binary reward carries at most one bit, and exactly one only when the pass rate is 50%. Toby Ord runs this accounting [1] out to frontier-length tasks and estimates that RL gives a model at most a thousandth to a millionth as much information to learn from per hour of training as pretraining.
Yet, as Beren Millidge points out [2], RL on a language model is strikingly sample efficient. It often takes a model from barely doing a task to doing it reliably in hundreds to thousands of steps, with gains visible after as few as ten. Ten steps of pretraining achieve essentially nothing. How is the bit-starved method the fast one?
The count is right, but it measures how many bits the reward carries, not how many of them bear on the task, or how many the task needs.
Take how many bits bear on the task first. A token's pretraining loss is about predicting that token, and most tokens in a rollout have nothing to do with whether it eventually succeeds, so measured against the task their gradients are mostly noise that swamps the few that matter. The reward on a checkable task doesn't have that problem: it is the objective itself, so every bit it carries points toward succeeding, trading fewer bits for a far higher signal-to-noise ratio. That's what makes Millidge's observation telling — fine-tuning a model on its own successful rollouts, which puts a loss on every token of the same traces, does worse than policy gradients, the opposite of what a bit count alone would predict.
The other half is how many bits the task needs, and a strong base is already most of the way there. RL-trained models beat their base models at pass@1, but given many samples the base models solve more problems [3]: RL is mostly raising the probability of reasoning the base can already produce, not teaching it new reasoning. That's why language-model RL starts from a pretrained model rather than from scratch — the from-scratch case, as with the Atari agents, was notoriously sample-hungry, because learning a language from nothing takes far more bits than a pass/fail signal can carry.
References
The Extreme Inefficiency of RL for Frontier Models [link]
Ord, T., 2025.How Can LLM RL Work Despite Information-Theoretic Inefficiency [link]
Millidge, B., 2026.Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? [PDF]
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S. and Huang, G., 2025. arXiv. DOI: 10.48550/ARXIV.2504.13837