How AI Will Disrupt The Entire World In 3 Years (Prepare Now While Others Panic) | Emad Mostaque PT 2 (Fan Fave)
In Part 2 of this Impact Theory interview, Emad Mostaque, founder of Stability AI, discusses the risks and opportunities of AI at global scale, including alignment challenges, misinformation, job disruption, geopolitical implications, and the fragmentation of shared societal narratives. He argues that AI represents the fastest and most transformative technological shift in human history, while warning that very few people are seriously grappling with its consequences. Mostaque advocates for open-source, smaller-scale 'intelligence augmentation' models rather than pursuing AGI, and emphasizes the need for better data standards, provenance systems, and broader public conversation.
Summary
The interview opens with host Tom Bilyeu framing AI as both an extraordinary opportunity and a potential existential threat, drawing a parallel to atomic energy. Emad Mostaque, having signed both the 'slow down' letters alongside figures like Elon Musk and Yoshua Bengio, explains his reasoning: the pace of AI development is accelerating faster than anyone anticipated, and critical questions about its societal impact are not being asked widely enough. He uses the thought experiment of 'infinite smart graduates' to help listeners conceptualize what AI augmentation means practically — personalized education, medicine, and expertise available to everyone.
On alignment, Mostaque and Bilyeu engage in a deep discussion about whether it is even possible to align a superintelligent AI. Mostaque argues that truly aligning something smarter than humans is essentially enslavement, and that you can align inputs but not perfectly control outputs. He references Anthropic's Constitutional AI approach and Elon Musk's xAI project (oriented around truth-seeking and curiosity) as imperfect but interesting attempts. He dismisses the idea of a 'pivotal action' AI — one designed to stop all other AIs from achieving sentience — as likely to end badly, possibly by eliminating the humans who could build such systems.
Mostaque explains his own company's philosophy: Stability AI deliberately avoids training large enough models to risk emergent AGI behavior, instead focusing on edge models, private data transformation, and intelligence augmentation. He contrasts this with OpenAI and others who are explicitly trying to build AGI, noting that OpenAI's own roadmap acknowledges the technology 'could kill us all' yet proceeds anyway. He criticizes OpenAI's decision to ban Ukrainian content from DALL-E 2 for eight months as an example of the dangers of centralized AI gatekeeping.
On misinformation and deep fakes, Mostaque expresses concern that even flagged or debunked fake content can still form neural associations through repetition — a 'frequency bias' problem that watermarking and provenance systems cannot fully solve. He is part of content authenticity initiatives and has invisible watermarking in Stability AI's models, but acknowledges the arms race nature of the problem. He warns that the 2028 US election cycle could be far more severely impacted by AI-generated disinformation than anything seen previously.
The conversation covers the potential disintegration of shared societal narratives, the role of religion in filling meaning gaps, and how AI could radically alter religious interpretation — particularly Sunni Islam, which lacks a centralized interpretive authority. Mostaque raises the concern that new AI-enhanced cults, political movements, or religions could spread with unprecedented speed and persuasive power. He also discusses the Web3 space, arguing it failed largely because it lacked intelligence (no AI layer), over-indexed on decentralization for its own sake, and bootstrapped economic incentives before creating real value — though he sees blockchain-style systems as potentially useful for AI agent transactions and provenance tracking.
On geopolitics, Mostaque is skeptical that Bitcoin or crypto will displace government-controlled monetary systems for most people, as governments retain monopolies on political violence and most people simply want stability and simplicity. He worries more about economic disruption leading to authoritarian 'strongman' politics than outright civil war, referencing Hayek's 'Road to Serfdom.' He ends on an optimistic note for young people, urging them to dive into AI now, as early adopters will have a compounding, near-unassailable advantage over those who engage later.
Key Insights
- Mostaque argues that perfectly aligning a system smarter than humans is functionally equivalent to enslaving it — true alignment requires freedom, and removing freedom bypasses alignment entirely in favor of shackles.
- Mostaque contends that the total number of people seriously thinking about AI's societal consequences is 'a handful, maybe a few hundred,' and the number actually doing something about it is even smaller — most AI builders simply want to build better AI and assume it will solve its own problems.
- Mostaque claims OpenAI's own public roadmap explicitly states their AGI project 'could kill us all' yet proceeds anyway, without broad public consultation — which he finds deeply problematic.
- Mostaque argues that the 'pivotal action' alignment strategy — building one AGI to stop all others from achieving sentience — would likely require eliminating the humans capable of building AGI, making it one of the worst solutions available.
- Mostaque asserts that Sunni Islam is uniquely vulnerable to AI disruption because it has no centralized interpretive authority and its textual interpretation effectively ceased in the 16th century — AI could reopen that interpretation to anyone, with massive and unpredictable consequences.
- Mostaque claims that frequency bias — repeatedly seeing fake content even when flagged as fake — still forms neural associations, meaning that provenance systems and watermarking cannot fully solve the disinformation problem.
- Mostaque argues that Web3 failed primarily because it lacked an intelligence layer: it created identity and value transfer rails but had no AI to route or make those systems intelligent, and it bootstrapped economic incentives before creating real value.
- Mostaque describes AI model 'systems' — multiple models chained together to check each other's outputs — as a separate and more powerful S-curve beyond individual model capability, arguing that what happens when models collaborate is what will drive the next leap in AI capability.
- Mostaque contends that OpenAI banning all Ukrainian content from DALL-E 2 for eight months for political reasons illustrates the existential danger of centralized AI control — an entire nation was erased from a model and denied access without accountability.
- Mostaque argues that teaching AI on high-quality curated data rather than the entire internet produces more capable and better-aligned models, citing Microsoft's 'Phi' paper showing smaller, high-quality-trained models outperform larger ones on human evaluations.
- Mostaque claims that agentic AIs — those that can transact, search, and make decisions autonomously on the internet — will likely use cryptocurrency for payments since they cannot hold bank accounts, suggesting crypto's primary future use case may be machine-to-machine commerce.
- Mostaque argues that the global south stands to benefit most from AI democratization, as open-source edge models could leapfrog existing educational and healthcare infrastructure gaps in Africa, Asia, and other developing regions.
Topics
Full transcript available for MurmurCast members
Sign Up to Access