The data black hole at the center of AI

The speaker opens by defining intelligence partly in terms of sample efficiency — how much data is needed to become competent in a domain — and argues that AI has made little progress on this metric. Instead, progress has come primarily from expanding and improving training data distributions, with reinforcement learning (RL) serving as a form of synthetic data generation where compute is used to identify high-quality rollouts.

The speaker emphasizes how extraordinarily task-specific and voluminous the human expert data required for AI training is, pointing to job listings at data labeling companies like Scale AI and Surge as evidence. Each skill may require hundreds of experts generating examples, rubrics, and chain-of-thought explanations. This data industry already generates billions in annual revenue.

To illustrate the sample efficiency gap, the speaker offers several comparisons: humans accumulate roughly 200 million tokens by adulthood, while frontier models train on tens to hundreds of trillions — a roughly million-fold difference. Similarly, humans can learn to teleoperate robots in hours, while AI requires millions of demonstration hours and still struggles with complex open-ended tasks. A teenager learns to drive in ~20 hours of practice, whereas autonomous vehicle systems require three to four orders of magnitude more data.

The speaker then addresses three common objections. First, the evolutionary pre-training argument — that evolution effectively pre-trained humans — is dismissed by noting the human genome is only 3GB with little coding capacity, suggesting evolution tuned hyperparameters rather than encoding network weights. Further, even pre-trained AI models still require massive data for each new marginal skill, unlike humans. Second, the multimodal sensory data objection is countered by noting that blind and deaf individuals still achieve general intelligence, suggesting sensory tokens are not the key driver of human intelligence. Third, the scaling argument — that larger models might close the sample efficiency gap — is refuted using Chinchilla scaling law equations, which show that even infinite parameters would only reduce data requirements by a factor of ~10, far short of the thousands-to-millions-fold gap that exists.

Despite these inefficiencies, the speaker argues that AI can still be economically viable for automating white-collar work. Common professional tasks can be brought into the training distribution, and the massive parallelism and scale of AI deployment makes even highly inefficient training worthwhile. The speaker notes that some jobs, like software engineering, may require significant out-of-distribution reasoning, and speculatively suggests demand for human software engineers may actually increase by 2027 due to AI complementarity.

Finally, the speaker teases a future discussion about whether current AI systems — lacking human-level sample efficiency — could nonetheless accelerate AI research enough to eventually solve the sample efficiency problem itself, cautioning that most thinking about intelligence explosions is too binary and lacks nuance about what progress would actually look like on top of current LLM architectures.