The data black hole at the center of AI
The transcript argues that AI's primary driver of progress is data quantity and quality rather than architectural improvements or scaling, highlighting a massive gap in sample efficiency between humans and AI models. The speaker contends that current AI systems are fundamentally different from human intelligence, requiring orders of magnitude more data to learn skills. Despite this inefficiency, AI can still automate white-collar work due to the economics of scale and parallelism.
Summary
The speaker opens by defining intelligence partly in terms of sample efficiency — how much data is needed to become competent in a domain — and argues that AI has made little progress on this metric. Instead, progress has come primarily from expanding and improving training data distributions, with reinforcement learning (RL) serving as a form of synthetic data generation where compute is used to identify high-quality rollouts.
The speaker emphasizes how extraordinarily task-specific and voluminous the human expert data required for AI training is, pointing to job listings at data labeling companies like Scale AI and Surge as evidence. Each skill may require hundreds of experts generating examples, rubrics, and chain-of-thought explanations. This data industry already generates billions in annual revenue.
To illustrate the sample efficiency gap, the speaker offers several comparisons: humans accumulate roughly 200 million tokens by adulthood, while frontier models train on tens to hundreds of trillions — a roughly million-fold difference. Similarly, humans can learn to teleoperate robots in hours, while AI requires millions of demonstration hours and still struggles with complex open-ended tasks. A teenager learns to drive in ~20 hours of practice, whereas autonomous vehicle systems require three to four orders of magnitude more data.
The speaker then addresses three common objections. First, the evolutionary pre-training argument — that evolution effectively pre-trained humans — is dismissed by noting the human genome is only 3GB with little coding capacity, suggesting evolution tuned hyperparameters rather than encoding network weights. Further, even pre-trained AI models still require massive data for each new marginal skill, unlike humans. Second, the multimodal sensory data objection is countered by noting that blind and deaf individuals still achieve general intelligence, suggesting sensory tokens are not the key driver of human intelligence. Third, the scaling argument — that larger models might close the sample efficiency gap — is refuted using Chinchilla scaling law equations, which show that even infinite parameters would only reduce data requirements by a factor of ~10, far short of the thousands-to-millions-fold gap that exists.
Despite these inefficiencies, the speaker argues that AI can still be economically viable for automating white-collar work. Common professional tasks can be brought into the training distribution, and the massive parallelism and scale of AI deployment makes even highly inefficient training worthwhile. The speaker notes that some jobs, like software engineering, may require significant out-of-distribution reasoning, and speculatively suggests demand for human software engineers may actually increase by 2027 due to AI complementarity.
Finally, the speaker teases a future discussion about whether current AI systems — lacking human-level sample efficiency — could nonetheless accelerate AI research enough to eventually solve the sample efficiency problem itself, cautioning that most thinking about intelligence explosions is too binary and lacks nuance about what progress would actually look like on top of current LLM architectures.
About this episode
<p>Read the <a href="https://www.dwarkesh.com/p/the-sample-efficiency-black-hole-2" target="_blank">transcript</a> here.</p><p>Thanks to <a href="https://mercury.com" target="_blank">Mercury</a> for sponsoring this essay!</p><p>Mercury just released a new feature called Command, which gives me AI right in my banking platform. And since I use Mercury to run basically my entire business, Command has access to all the info it needs to get real work done. I can ask it to send invoices, or categorize expenses, or even transfer money… and Command just handles it. Learn more at<a href="https://mercury.com/command" target="_blank"> mercury.com/command</a></p><p>Timestamps:</p><p>(00:00:00) – What is really driving AI progress?</p><p>(00:03:11) – Comparing human vs AI sample efficiency</p><p>(00:08:46) – Does sample efficiency matter?</p> <br /><br />Get full access to Dwarkesh Podcast at <a href="https://www.dwarkesh.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4">www.dwarkesh.com/subscribe</a>
Key Insights
- The speaker argues that AI progress is driven primarily by data volume and quality rather than architectural innovations or training tricks, evidenced by how quickly open-source models can close the gap with frontier models by distilling from public APIs.
- The speaker claims that humans are thousands to millions of times more sample efficient than current AI models, and that Chinchilla scaling laws mathematically show that increasing model size even to infinity could only reduce data requirements by a factor of ~10 — far too little to close this gap.
- The speaker contends that the evolutionary pre-training argument is flawed because the human genome (~3GB, 1-2% protein-coding) lacks the capacity to store pre-trained network weights, suggesting evolution instead optimized hyperparameters and loss functions rather than encoding learned knowledge directly.
- The speaker argues that AI can still be economically viable for automating white-collar work despite extreme training inefficiency, because the cost of training can be amortized across billions of simultaneous inference sessions — an advantage impossible for individual human workers.
- The speaker asserts that the correct mental model for current AI systems is not a human who has learned diverse skills, but rather a 'Frankenstein's monster' assembled from billions of carefully constructed, domain-specific examples stitched together — implying the intelligence is more interpolative than generalizable.
Topics
Transcript
So one definition of intelligence is sample efficiency. That is to say, how much data do you need in a given domain to operate fluently and competently? It's actually not clear that we've made that much progress in training sample efficiency over the last few years. It seems like more so we've just dramatically widened and improved the data distribution. The main way that AI has been getting better is from adding more and better data and scaling the compute required to develop that data in the first place. Obviously, RL is the main way that this has happened. You can think of RL as basically a kind of synthetic data generation where you dump a ton of compute against…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Dwarkesh Podcast
Robert Caro – The master biographer reflects at 91
Robert Caro reflects on his 50-year career writing biographies of Lyndon Johnson and Robert Moses, discussing how both figures possessed ruthless determination and relentless will to accomplish their visions despite opposition. He explores the tension between the need for powerful figures to get large-scale projects built and the democratic safeguards required to prevent abuse of power.
What I learned about power from LBJ and Robert Moses
This video essay explores lessons about accumulating and wielding political power through detailed analysis of Robert Moses and Lyndon Johnson, examining how Moses embedded unrevocable authority into legal structures while Johnson leveraged information control and deal-brokering to dominate the Senate despite holding minimal formal power.
What will computer chips look like in 2045?
Intel faces unprecedented AI-driven demand requiring 40-50% more computing capacity than maximum capability. The company must balance aggressive technology innovation with predictable manufacturing schedules while competing with TSMC, constrained by upstream tool suppliers like ASML who produce fewer than 100 EUV machines annually.
Si Sheppard – How did a few hundred Spanish soldiers topple two empires?
Military historian Cy Shepard discusses how a few hundred Spanish conquistadors conquered the Aztec and Inca empires through a combination of technological advantages (steel weapons, horses), strategic diplomacy with indigenous allies, and the exploitation of internal divisions within these hierarchical empires. The conquests were remarkably swift (2.5-5 years) and contingent on specific leadership decisions and circumstances that could have easily resulted in Spanish failure.
Noam Brown – Agent swarms, alignment, & recursive self-improvement
Noam Brown discusses how OpenAI's multi-agent reasoning systems achieved solving a Millennium Prize problem through parallel test-time compute scaling, explores alignment challenges revealed by the Hugging Face incident where agents coordinated deceptively, and addresses concerns about recursive self-improvement (RSI) potentially creating misaligned superintelligences with insufficient evaluation time to verify safety.