The Problem With How LLMs Generate Text - Grant Sanderson
Grant Sanderson explains that autoregressive text generation in LLMs is fundamentally misaligned with human writing processes. Rather than composing with foresight and deep connections, LLMs are forced to predict tokens sequentially with memory wiped between predictions, making it difficult for them to form the unlikely but substantive connections that characterize good writing.
Summary
Sanderson uses an analogy to illustrate the oddness of autoregressive generation: imagine an intelligent person locked in a box who can only interact by receiving slips of paper asking them to predict what comes next, then having their memory wiped repeatedly. The resulting essay would be poor despite the person's intelligence, because sequential prediction is fundamentally different from compositional thinking. He argues that LLMs become slaves to immediate context—drawing connections within a narrow field—but the truly valuable connections are inherently unlikely and unpredictable as next tokens. While reinforcement learning can improve performance in some ways, there's no specific incentive mechanism driving the model toward making unlikely but substantive connections when the vast majority of unlikely connections aren't the statistically predictable next token. Sanderson concludes that this suggests intelligent capability may be locked inside the model, but autoregression is simply a weird constraint on how that intelligence can interact with and produce text.
Key Insights
- Autoregressive generation fundamentally differs from human compositional writing because humans plan and think through ideas holistically, while LLMs predict tokens sequentially with no memory between steps
- LLMs become enslaved to immediate context when answering questions, drawing connections narrowly within a particular field rather than making the substantive cross-domain connections that characterize good writing
- The most valuable and substantive connections in writing are inherently unlikely as next-token predictions, creating a misalignment between what makes good writing and what autoregressive models are optimized to predict
- Reinforcement learning improvements to LLMs lack specific incentive mechanisms that would push the model toward generating unlikely-but-substantive connections, since these connections contradict the statistical likelihood objective
- There may be significant intelligence locked inside LLMs, but the autoregressive interaction mechanism is an inherently limiting constraint on how that intelligence can be expressed through text generation
Topics
Transcript
[0:00] Auto regression is actually like a really really weird [music] way to produce stuff if you think about it. Like you're an intelligent person. Imagine I locked you in a box and then the only way that you have of interacting with the world is that you receive a slip of paper and then someone says, "Can you like predict what will come next?" And then you predict what will come next and then your memor is [music] wiped. Imagine that was done a whole bunch and then what comes out on the other end? They're like, "Look at this essay that [music] you wrote." You might look at that and be like, "This is awful. That's not the…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Dwarkesh Patel
How a Random Lunch Led Physics into the Riemann Hypothesis - Grant Sanderson
The discussion highlights a connection between number theory and random matrix theory through the collaboration of Hugh Montgomery and Freeman Dyson, showcasing the interdisciplinary nature of mathematical research. Their findings on the Riemann Hypothesis and the zeros of the Riemann zeta function hint at a deeper similarity between seemingly unrelated fields.
8 Predictions for the Era of Continual Learning
The speaker outlines eight major predictions for how AI systems with continual learning capabilities will transform the industry, regulatory frameworks, technical alignment approaches, market dynamics, and competitive landscapes. Continual learning—where models improve from real-world deployment experience rather than remaining static after training—fundamentally changes assumptions about AI safety, deployment, and business models.
The Skill Great Teachers Have That LLMs Completely Lack - Grant Sanderson
Grant Sanderson discusses a critical limitation of LLMs compared to great human teachers: the inability to reframe or redirect flawed student thinking while validating the creative reasoning behind it. Great teachers can recognize when students approach problems incorrectly and guide them toward better frameworks without dismissing their underlying logic.
The Real Advantage AI Has Over Human Geniuses - Grant Sanderson
Grant Sanderson discusses how AI systems can overcome cognitive biases by systematically adopting different contexts and approaches, using multiple agents with conflicting objectives. He illustrates this with an IMO problem where the elegant intuitive solution was incorrect, and argues that AI's ability to deliberately introduce entropy and diversity could be a key advantage over human thinking patterns.
Why smarter AI models could drive up compute prices 10x
As AI labs like Anthropic scale revenue 10x year-over-year while compute capacity only grows 3x, compute prices must rise significantly to bridge the gap. The speaker argues that as AI models become more capable, they can monetize the same compute at much higher rates, potentially driving prices up 10-15x, creating winner-take-most dynamics and pricing out current AI applications.