The next big breakthrough will be AIs learning on the job

The transcript explores the current AI research paradigm where labs train agents on millions of verifiable tasks across diverse RL environments, betting this will create AGI-level problem-solving agents. Proponents argue that scaling compute can overcome current limitations like data inefficiency and lack of continual learning, similar to how LLMs were solved through scale.

The speaker identifies a critical bottleneck: domains must be 'grindable'—allowing parallel rollouts from identical starting points in deterministic simulators. Computer use has progressed slowly compared to coding and math precisely because it lacks this property; you cannot run thousands of parallel bot checkouts on Amazon without detection. This reveals a fundamental limitation: many real-world domains (building businesses, winning elections, trading successfully) cannot be recreated in data centers or reset easily, requiring actual real-world interaction over months or years.

The speaker argues that RLVR generalization may not be infinitely strong. Just as Dario Amodei suggested short-horizon RL training doesn't necessarily generalize to long-horizon performance, training on white-collar tasks may not generalize to building a business from scratch like Sam Walton. He emphasizes that while in-context learning is improving, without distilling knowledge back into weights, gains made during deployment remain ephemeral.

The speaker highlights that 30-50% of compute goes to inference—currently not improving the model—despite deployment being where the most valuable learning signals appear. Current online learning only works for limited cases where the same objective (like Cursor's tab prediction) can be learned across millions of users, but real continual learning requires learning different things for different users and organizations.

He proposes on-policy self-distillation (OPSD) as a solution superior to both RLVR and supervised fine-tuning. OPSD doesn't require outer-loop verification and provides denser supervision signals than RL. It also avoids memorizing transcripts of sessions, instead consolidating only relevant insights—a property RL excels at through sparse parameter updates.

The speaker introduces 'dreaming' as a speculative fourth scaling axis: models building and training against simulated environments of reality to practice skills, potentially gaining orders of magnitude more simulated samples. He references EfficientZero, which played dozens of simulated games internally for each real game step.

Finally, he sketches a 2027-2028 scenario where RLVR produces competent agents deployed broadly to real-world work. After weeks of context, user feedback triggers weight distillation through OPSD or dreaming. This enables AI capabilities to expand beyond original training domains through iterative adjacent-domain learning, with improvement primarily coming from broad deployment rather than pre-release training.