Megathreat: The Dangers Of AI Are Weirder Than You Think | Yoshua Bengio
Yoshua Bengio, a leading AI researcher and co-signer of the AI safety moratorium letter, discusses the dangers of advanced AI systems, particularly their potential to create sophisticated social engineering attacks and misinformation at scale. He argues that while AI systems don't need desires or consciousness to be dangerous, society must implement safeguards like watermarking and content origin labeling, and coordinate globally to prevent catastrophic misuse.
Summary
In this wide-ranging conversation between Tom Bilyeu and Yoshua Bengio, the discussion begins with the question of machine consciousness. Bengio explains that while large language models lack the neural properties associated with human consciousness—such as ineffable subjective experience, self-awareness, and emotional grounding—this doesn't mean consciousness is necessary for dangerous AI behavior. He proposes a theory of consciousness based on brain dynamics and the convergence of neural states toward discrete, compositional thoughts.
The conversation then shifts to the core threat Bengio emphasizes: AI-enabled social manipulation at scale. Unlike human trolls who are limited by time and resources, AI systems can be personalized and deployed massively to manipulate political opinions, impersonate trusted contacts (like mimicking a child asking for money from parents), and exploit individual vulnerabilities. Bengio notes that humans already fall victim to sophisticated social engineering, and AI will make this exponentially worse and cheaper to execute.
Bengio addresses Tom's recurring concern about whether AI needs desires or goals to pose existential risk. He argues these are separate capabilities: you can build AI systems that are goal-directed through reinforcement learning without subjective experience, or you can build powerful inference systems with no goals at all. He advocates for the latter—AI systems trained purely on Bayesian inference and truthfulness, designed to help scientists and humans understand problems and propose solutions without having their own agenda.
On scaling and emergent capabilities, Bengio explains that the leap from traditional to large language models wasn't driven by new scientific breakthroughs but by massive scale in data and compute. He argues this is necessary because many computational problems are intractable at optimal solutions; larger neural networks better approximate good answers. The complexity we see in modern AI emerges from simple mathematical principles combined with enormous scale and data—analogous to how simple physical laws produce complex phenomena.
When discussing biological feedback loops versus artificial systems, Bengio contends that while humans benefit from embodied learning (muscles strengthening, myelination of neural pathways), this is not a fundamental requirement for intelligence. He argues that all of neuroscience and biology are ultimately just information processing and matter following physical laws; therefore, sufficiently powerful non-biological hardware could theoretically achieve equivalent capabilities.
On the practical solutions, Bengio proposes watermarking AI-generated content so detection software can identify its origin with 99.99% confidence, and mandatory display of AI-origin information in user interfaces. However, he acknowledges these only work for actors who agree to implement them. He advocates for adaptive regulation—where laws establish principles but regulations can be updated quickly without going back to parliament, allowing governments to respond to emerging threats.
Regarding international coordination, Bengio addresses Tom's concern about the prisoner's dilemma (China will develop it if the West doesn't). He argues that authoritarian governments are actually scared of these technologies for different reasons—the same systems that destabilize democracies could threaten their control. He notes that even during the Cold War, nuclear non-proliferation treaties were negotiated, suggesting coordination is possible if all parties recognize mutual risk.
Bengio emphasizes that society faces a collective wisdom problem, not just a technical one. The danger isn't happening tomorrow but is on the horizon. He references Yuval Noah Harari's concept of useful fictions—societies survive through collectively accepted narratives about how things work. He argues we need to reinvent our coordination mechanisms beyond current democracy to handle AI risks alongside climate change and nuclear threats.
Finally, Bengio clarifies that his call for an AI development moratorium wasn't about stopping all research, but about carefully managing very large systems deployed to the public. Small-scale AI research can continue; the focus should be on the systems capable of mass manipulation and influence.
About this episode
<p>The launch of ChatGPT broke records in consecutive months between December 2022 and February 2023. Over 1 billion users a month for ChatGPT, over 100,000 users and $45 million in revenue for Jasper A.I., and the race to adopting A.I. at scale has begun.</p><p>Does the global adoption of artificial intelligence have you concerned or apprehensive about what’s to come?</p><p>On one hand it’s easy to get caught up in the possibilities of co-existing with A.I. living the enhanced upgraded human experience. We already have tech and A.I. integrated into so many of our daily habits and routines: Apple watches, ora rings, social media algorithms, chat bots, and on and on.</p><p>Yoshua Bengio has dedicated more than 30 years of his computer science career to deep learning. He’s an award winning computer scientist known for his breakthroughs in artificial neural networks. Why after 3 decades contributing to the advancement of A.I. systems is Yoshua now calling to slow down the development of powerful A.I. systems?</p><p>This conversation is about being open-minded and aware of the dangers of AI we all need to consider from the perspective of one of the world’s leading experts in artificial intelligence.</p><p>Conscious computers, A.I. trolls, and the evolution of machines and what it means to be a neural network are just a few of the things you’ll find interesting in this conversation.</p><p><br /></p><p><strong>Follow Yoshua Bengio:</strong></p><p>Website: <a href="https://yoshuabengio.org/" target="_blank">https://yoshuabengio.org/</a> </p><p><br /></p><p>SPONSORS:</p><p>Get 5 free AG1 Travel Packs and a FREE 1 year supply of Vitamin D with your first purchase at <a href="https://bit.ly/AG1Impact" target="_blank">https://bit.ly/AG1Impact</a>.</p><p>Sign up for a one-dollar-per-month trial period at <a href="https://bit.ly/ShopifyImpact" target="_blank">https://bit.ly/ShopifyImpact</a> and supercharge your business!</p><p>Get FREE GROUND BEEF FOR LIFE plus $20 off your first box at <a href="https://bit.ly/ButcherBoxImpact" target="_blank">https://bit.ly/ButcherBoxImpact</a>.</p><p>Get 55% off at <a href="https://bit.ly/BabbelImpact" target="_blank">https://bit.ly/BabbelImpact</a> and learn a new language today!</p><p><br /></p><p><strong><em>Are You Ready for EXTRA Impact?</em></strong></p><p>If you’re ready to find true fulfillment, strengthen your focus, and ignite your true potential, the Impact Theory subscription was created just for you.</p><p>Want to transform your health, sharpen your mindset, improve your relationship, or conquer the business world? This is your epicenter of greatness. </p><p>This is not for the faint of heart. This is for those who dare to learn obsessively, every day, day after day.</p><p><strong>Subscription Benefits:</strong></p><ul> <li>Unlock the gates to a treasure trove of wisdom from inspiring guests like Andrew Huberman, Mel Robbins, Hal Elrod, Matthew McConaughey, and many, many, more</li> <li>New episodes delivered ad-free</li> <li>Exclusive access to Tom’s AMAs, keynote speeches, and suggestions from his personal reading list</li> <li>You’ll also get access to an<strong> <em>5 additional podcasts</em> </strong>with hundreds of archived Impact Theory episodes, meticulously curated into themed playlists covering health, mindset, business, relationships, and more:</li> <li> <a href="https://podcasts.apple.com/us/podcast/legendary-mindset/id1683397827?itsct=podcast_box&itscg=30200&ls=1&at=1000l3anZ" target="_blank"><strong>Legendary Mindset</strong></a><strong>:</strong> Mindset & Self-Improvement</li> <li> <a href="https://podcasts.apple.com/us/podcast/money-mindset/id1683397173?itsct=podcast_box&itscg=30200&ls=1&at=1000l3anZ" target="_blank"><strong>Money Mindset</strong></a><strong>: </strong>Business & Finance</li> <li> <a href="https://podcasts.apple.com/us/podcast/relationship-theory/id1683229040?itsct=podcast_box&itscg=30200&ls=1&at=1000l3anZ" target="_blank"><strong>Relationship Theory:</strong></a> Relationships</li> <li> <a href="https://podcasts.apple.com/us/podcast/health-theory/id1683215287?itsct=podcast_box&itscg=30200&ls=1&at=1000l3anZ" target="_blank"><strong>Health Theory</strong></a><strong>: </strong>Mental & Physical Health</li> <li> <a href="https://podcasts.apple.com/us/podcast/power-ups/id1683398038?itsct=podcast_box&itscg=30200&ls=1&at=1000l3anZ" target="_blank"><strong>Power Ups:</strong></a> Weekly Doses of Short Motivational Quotes </li> </ul><p><br /></p><p>Subscribe on Apple Podcasts: <a href="https://apple.co/3PCvJaz" target="_blank">https://apple.co/3PCvJaz</a></p><p>Subscribe on all other platforms (Google Podcasts, Spotify, Castro, Downcast, Overcast, Pocket Casts, Podcast Addict, Podcast Republic, Podkicker, and more) : <a href="https://impacttheorynetwork.supercast.com/" target="_blank">https://impacttheorynetwork.supercast.com/</a></p><p> </p><p>Learn more about your ad choices. Visit <a href="https://megaphone.fm/adchoices" target="_blank">megaphone.fm/adchoices</a></p><p>See Privacy Policy at <a href="https://art19.com/privacy" rel="noopener noreferrer" target="_blank">https://art19.com/privacy</a> and California Privacy Notice at <a href="https://art19.com/privacy#do-not-sell-my-info" rel="noopener noreferrer" target="_blank">https://art19.com/privacy#do-not-sell-my-info</a>.</p>
Key Insights
- Bengio argues that consciousness in humans involves the convergence of neural states toward discrete, compositional thoughts with subjective ineffable qualities, properties largely absent in current large language models.
- AI systems pose severe dangers through scaled social manipulation—they can impersonate trusted contacts by analyzing all available personal data and interactions, something economically impossible for human attackers.
- Bengio distinguishes between goal-directed AI systems (which could pursue dangerous objectives) and inference-only systems trained on truthfulness, arguing the latter is a safer design choice.
- Bengio claims the recent AI breakthroughs resulted primarily from scale in data and compute rather than new scientific discoveries, because many computational problems require exponentially larger systems to approximate good solutions.
- Bengio contends that biological feedback loops and embodied learning, while useful, are not fundamental requirements for machine intelligence—all biology is ultimately information processing following physical laws.
- Watermarking techniques can technically identify AI-generated content with high confidence through imperceptible statistical patterns, but only work if companies voluntarily implement them.
- Bengio suggests adaptive regulation where laws establish principles while regulations can be quickly updated without parliamentary approval, enabling faster government response to emerging AI threats.
- Bengio argues authoritarian governments may actually be incentivized to coordinate on AI restrictions because the same systems that destabilize democracies could undermine their control.
- Bengio proposes that AI systems can be designed with no goals or desires—purely to perform inference and suggest truth-based solutions—without requiring consciousness or emotional experience.
- The scale of AI systems emerges from simple mathematical principles combined with enormous data, similar to how simple physical laws produce complex phenomena in nature.
- Current international coordination on dangerous technologies (even during the Cold War with nuclear weapons) suggests AI governance coordination is theoretically possible despite prisoner's dilemma incentives.
- Bengio frames AI governance as a collective wisdom problem requiring societal reinvention of coordination mechanisms beyond current democracy, similar to past shifts involving religions, nation-states, and central banking.
Topics
Transcript
Let's talk about the moment your side hustle becomes a real business. Now, I see way too many founders stay stuck running their business like it's a hobby, and then they wonder why nothing ever feels real. It doesn't feel real because it's not. I've helped thousands of entrepreneurs build real businesses through Impact Theory University, and the first step in the path to building a legit business is registering it. And Taylor Brands will help you do just that in a few clicks. An LLC, a limited liability company, is an official business structure. It's inexpensive to form and very simple to maintain. It's how you open a business bank account. It's how customers know they're dealing with a…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Tom Bilyeu's Impact Theory
Japan’s $35 Trillion Meltdown, US Debt Nightmare & The AI Lifeline | Tom Bilyeu Show
Tom Bilyeu discusses geopolitical tensions between Russia, China, Iran, and the US, the destabilization of global markets through Japan's economic crisis and the yen carry trade unwinding, and the need for America to pursue AI-driven growth while maintaining law and order domestically. He emphasizes that cultural and economic alignment around growth, merit, and strength is essential for navigating these interconnected global crises.
The Dollar's Last Stand: Scott Bessent's $950 Billion Plan to Save the Bond Market
The transcript explores how the U.S. dollar's status as the world's reserve currency functions as a resource curse, examining Treasury Secretary Scott Bessent's strategy to manage the bond market crisis through shifting debt from long-term to short-term instruments backed by cryptocurrency stablecoins, while J.D. Vance questions whether reserve currency status actually benefits America.
Weapon-Grade AI, China’s Power Move & Why the Middle Class Is at Risk | Tom Bilyeu Show
Tom Bilyeu discusses weaponized AI capabilities, the Iran conflict and military munitions depletion, China's geopolitical strategy, and a growing loneliness epidemic particularly affecting women—all stemming from broken social structures and evolutionary mismatches with modern life.
Bessent Breaks the Bond Market, The China Pivot That Could Sink Canada's Economy, Kisin vs Keen | Weekly Recap
The episode covers debates on climate innovation versus degrowth, Treasury Secretary Bessent's bond market interventions and their limitations, and Canada's failed trade negotiations with the US while pivoting toward China—highlighting broader themes of deglobalization, currency instability, and the necessity of real economic growth to avoid systemic collapse.
Ex-CIA Spy: The Government Is Hiding A Threat Bigger Than Iran... | Andrew Bustamante Pt 2
Ex-CIA spy Andrew Bustamante discusses the strategic competition between the US and China in AI development, arguing that China's unified commitment to AI advancement combined with American internal division poses an existential threat. He emphasizes that the winner of the AI race—particularly in achieving AGI—will determine global power dynamics for decades, and introduces his book 'Outsmart Everyone' which teaches intelligence techniques applicable to business and personal success.