Reacting to "Why AI is so smart but also so dumb?"
Andrej Karpathy discusses the evolution of AI from Software 1.0 to Software 3.0, explaining why LLMs excel in verifiable domains like code and math while struggling in others. He introduces the concepts of vibe coding versus agentic engineering, and argues that the entire internet needs to be rebuilt with agents in mind.
Summary
The video is a reaction to Andrej Karpathy's talk at Sequoia's annual AI event, where he explains the jagged nature of AI capabilities and the paradigm shift toward Software 3.0. Karpathy describes a clear inflection point in December 2024 where agentic coding tools stopped producing imperfect snippets and began reliably completing entire applications end-to-end, marking a fundamental change in how developers interact with AI.
Karpathy outlines his framework of Software 1.0 (explicit human-written rules), Software 2.0 (learned neural network weights trained on datasets), and Software 3.0 (LLMs as programmable computers where prompts serve as the programming language and the context window acts as RAM). He uses the OpenClaw installation example to illustrate how agent-native software design means providing outcomes and available tools rather than step-by-step instructions.
A central theme is verifiability as the key driver of AI capability. Karpathy explains that frontier labs train models using reinforcement learning with verification rewards, causing models to peak in domains where outputs can be easily verified — like code and math — while remaining rough in unverifiable domains. This explains the famous 'jaggedness' of AI, where a model can refactor a 100,000-line codebase but incorrectly advise walking to a car wash 50 meters away.
Karpathy distinguishes vibe coding (raising the floor — enabling non-engineers to build software) from agentic engineering (raising the ceiling — enabling professional engineers to maintain quality while moving dramatically faster using orchestrated agent swarms). He notes that taste, judgment, and orchestration remain human responsibilities, though he believes nothing fundamentally prevents models from eventually developing taste through better RL reward design.
The talk concludes with Karpathy expressing hope for more agent-first infrastructure, noting that virtually everything on the internet is still built for humans and needs to be rebuilt for agents. He ends with a profound observation: 'You can outsource your thinking but you can't outsource your understanding,' arguing that directing AI agents still fundamentally requires human understanding.
Key Insights
- Karpathy identifies December 2024 as a clear inflection point where agentic coding models stopped requiring correction and began reliably completing entire applications end-to-end, representing a fundamental shift rather than incremental improvement.
- Karpathy argues that in Software 3.0, the LLM acts as the CPU, the context window serves as RAM (short-term memory), and prompting replaces traditional programming — representing an entirely new computing paradigm rather than better software.
- Karpathy explains that frontier labs train LLMs using reinforcement learning with verification rewards, which causes models to develop jagged capabilities — excelling in verifiable domains like math and code while stagnating in domains where outputs cannot be easily verified.
- Karpathy uses the example of a state-of-the-art model simultaneously being able to refactor a 100,000-line codebase or find zero-day vulnerabilities, yet incorrectly advising a user to walk to a car wash 50 meters away, as evidence that AGI has not yet been achieved.
- Karpathy distinguishes vibe coding — which raises the floor by enabling anyone to build software regardless of technical knowledge — from agentic engineering, which raises the ceiling by allowing professional engineers to maintain quality bars while moving dramatically faster using orchestrated agents.
- Karpathy argues that taste and aesthetic judgment in AI code output currently suffers not because of any fundamental limitation, but because AI labs have not yet incorporated aesthetics into their reinforcement learning reward functions.
- Karpathy describes the 'animals vs ghosts' framing to argue that LLMs are not sentient beings with intrinsic motivation or curiosity shaped by evolution, but mathematical entities shaped purely by data and reward functions — a model he believes helps users interact with AI more competently.
- Karpathy cites a tweet that deeply influenced him — 'you can outsource your thinking but you can't outsource your understanding' — arguing that directing AI agents and determining what is worth building still requires fundamental human understanding that cannot be delegated.
Topics
Full transcript available for MurmurCast members
Sign Up to Access