AI researchers debate how close we are to recursive self-improvement

The discussion centers on whether AI will achieve recursive self-improvement (RSI) leading to superintelligence by 2036. The researchers identify several key bottlenecks preventing rapid takeoff. First, they emphasize the difficulty of specifying objectives for open-ended discovery tasks like paradigm shifts—while AI excels at optimization given clear objectives, determining *what* to optimize for remains hard. Second, they discuss how RL training on LLMs has been more successful than expected despite learning only ~1 bit per episode, attributing success to strong mid-training warm-starting, high signal-to-noise ratios compared to SFT, and horizon generalization across task lengths rather than broad domain transfer.

On data and scaling, the researchers estimate that pre-training improvements since 2019 stem from roughly 12x better data efficiency and 3.7x better architectures at small scales. They debate whether this continues at scale and whether data or architecture is the binding constraint. The consensus is that high-quality pre-training data is becoming scarce, pushing progress toward post-training RL and mid-training on synthetic reasoning data. Parameter scaling may plateau as inference efficiency becomes critical for RL rollouts, though this depends on whether compute or data becomes the bottleneck.

A major theme is the sim-to-real gap. Current training uses simulated environments in datacenters, but long-horizon real-world tasks—running a business, winning court cases, day trading—require genuine interaction and potentially continuous weight updates, not just transfer from simulation. They discuss continual learning challenges: naive fine-tuning causes catastrophic forgetting, while RL can preserve existing capabilities but limits how much new knowledge can be injected. They note that while labs can do continual mid-training cycles (e.g., releasing improved models every few months), truly continuous weight updates on individual deployed instances face technical barriers.

On consolidation, they argue distillation prevents winner-take-all outcomes because behaviors learned through RL are compressible; smaller models can match frontier models if given access to the right prompt distribution and deployment data. They discuss how Chinese companies extract useful training data through router services, giving them advantages in distillation that partially offset frontier labs' RL environment expertise.

Finally, they forecast timelines: drop-in remote workers for white-collar tasks in 1-3 years, 10x AI researcher productivity uplift in 2 years, and AI dominating all human experts in cognitive work within 3-5 years, with the latter approaching ASI-completeness due to requirements for learning new domains and handling non-stationary environments.