SN 1090: Black Hat - The Hidden Flaws in AI Security Nobody Saw Coming
Security Now episode 1090, recorded live at Black Hat 2026 with Steve Gibson, Paul Thurrott, and Richard Campbell, focuses on AI's impact on security, cybersecurity vulnerabilities, and the evolution of AI capabilities in both offensive and defensive contexts. The panel discusses how AI is revolutionizing code generation, vulnerability discovery, and the dual-use problem of AI knowledge.
Summary
This special live episode of Security Now from the Black Hat Conference features Steve Gibson alongside Windows Weekly hosts Paul Thurrott and Richard Campbell in a roundtable discussion format addressing listener questions about AI and security.
The conversation begins with personal accounts of AI adoption. Steve Gibson remains cautious, having used Claude for advisory purposes but not yet for code generation, citing age and preference for learning APIs directly. Leo Laporte describes his transformation beginning November 24, 2025, when Opus 4.5 released, moving from skepticism to active use of multiple AI agents (Claude, GPT-5.6, Deep Seek) running simultaneously on local hardware for projects including an 81,000-line sales system. Paul Thurrott uses AI selectively for hobby coding after struggling with Windows App SDK, while Richard Campbell explores both inside and outside AI models, working with teams using fully agentic development processes.
The panel discusses vulnerability discovery through AI, particularly the Cold Card wallet bug where a random number generator failure reduced entropy from 256 bits to 32 bits, remaining undetected for five years before a Chinese model (Kimi K3) discovered it, leading to approximately $100 million in Bitcoin theft. They emphasize that vulnerability discovery is now easier than fixing—AI can find exploits but fixing requires understanding architectural context that may exceed context windows for large codebases.
A critical security concern emerges regarding role confusion in LLMs. Lauren Weinstein's analysis reveals that models partition sessions using role tags (user, assistant, system) as mere formatting tricks rather than true security boundaries. Since everything to the model is "token soup," prompt injection succeeds because the model doesn't truly distinguish between instruction, data, and its own previous outputs. Content between tags influences role interpretation more than tags themselves.
The panel addresses the dual-use problem: AI trained on all knowledge cannot distinguish defensive from offensive questions, both requiring identical understanding. Current safeguards (refusals, classifiers, jailbreak resistance) don't remove knowledge from models. GRAM (Gradient Routed Auxiliary Modules), developed with Anthropic partnership, offers potential by partitioning restricted knowledge into removable neural slices, allowing version control without retraining massive models—solving the economics of training separate models for different use cases.
Discussing open-source intelligence and knowledge distribution, the panel argues that filtering knowledge is problematic and historically ineffective. They note China's open models and global accessibility of unconstrained versions, making knowledge restriction futile. The conversation touches on distillation accusations against Chinese models, suggesting such claims are unsubstantiated given timeline impossibilities.
The panel debates education's future, questioning traditional college value when AI can generate custom applications. They reference listener Andy Olson's experience creating 17 apps via Claude without professional programming background, and Jack Christensen's security contributions to Go SQL drivers, arguing AI democratizes development.
On AI's fundamental unreliability, they discuss how modern LLMs are probabilistic rather than deterministic, with temperature-injected randomness by design. Leo describes iterating on code with diminishing-returns thresholds (2% improvement minimum), and the challenge of context windows limiting holistic system understanding—Leo's 81,000-line project required 30-40 modular chunks specifically to manage this constraint.
They express concerns about gatekeeping: corporations restricting access to AI with defense capabilities mirrors historical control by priesthoods. Bitwarden's Agent Access SDK emerges as example of secure credential management for AI agents, though perfect solutions remain elusive.
About this episode
<p>At Black Hat Las Vegas, the Security Now crew digs into how AI is not just finding hidden software bugs but also fueling both groundbreaking innovation and alarming new exploits. When open models can launch surprise Bitcoin heists, who draws the line between forbidden knowledge and genuine progress?</p> <p>• Black Hat and DEF CON: Hacking Stories and Conference Culture<br /> • Zoox Ride-Hailing Hack and Over-the-Air Vulnerabilities<br /> • Autonomous Vehicles, AI, and the Security Implications<br /> • Hosts Share Personal Adoption and Use of AI Tools<br /> • AI-Powered Coding: From Hobbyists to Advanced Agency Chains<br /> • Local Models vs. Cloud AI: Privacy, Cost, and Control<br /> • App Development Democratized: Listeners Build Custom Solutions With AI<br /> • Code Generation, Testing, and Managing AI-Driven Project Cycles<br /> • AI's Role in Security: Vulnerability Discovery, Exploitation, and Patch Challenges<br /> • Technical Debt and the Race to Patch Decades-Old Bugs<br /> • The Dual-Use Dilemma: AI Tools for Both Attack and Defense<br /> • Guardrails, Model Partitioning, and the Fight Over Forbidden Knowledge<br /> • Open vs. Restricted AI: Global Models, Distillation, and Free Speech<br /> • LLM Security Weaknesses: Prompt Injection and Role Confusion Exposed<br /> • The Reliability Problem: Probabilistic AI and Non-Deterministic Software<br /> • AI Progress: Public Perception, Skepticism, and "Hogwash" Rebuttals<br /> • Reflections on AI's Fast Evolution and the Sci-Fi Reality Gap<br /> • Closing Thoughts: Tech Community, Listener Feedback, and the Future of Security Now</p> <p><strong>Hosts:</strong> <a href="https://twit.tv/people/steve-gibson">Steve Gibson</a>, <a href="https://twit.tv/people/leo-laporte">Leo Laporte</a>, <a href="https://twit.tv/people/richard-campbell">Richard Campbell</a>, and <a href="https://twit.tv/people/paul-thurrott">Paul Thurrott</a></p> <p>Download or subscribe to <em>Security Now</em> at <a href="https://twit.tv/shows/security-now">https://twit.tv/shows/security-now</a>.</p> <p>You can submit a question to <em>Security Now</em> at the <a href="https://www.grc.com/feedback.htm" target="_blank">GRC Feedback Page</a>.</p> <p>For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: <a href="https://www.grc.com/securitynow.htm" target="_blank">grc.com</a>, also the home of the best disk maintenance and recovery utility ever written <a href="https://www.grc.com/sr/spinrite.htm" target="_blank">Spinrite 6</a>.</p> <p><strong>Join Club TWiT for Ad-Free Podcasts!</strong><br /> Support what you love and get ad-free audio <em>and</em> video feeds, a members-only Discord, and exclusive content. Join today: <a href="https://twit.tv/clubtwit" rel="payment">https://twit.tv/clubtwit</a></p> <p><strong>Sponsors:</strong><ul> <li><a href="https://material.security" rel="sponsored" target="_blank">material.security</a></li> <li><a href="http://bitwarden.com/twit" rel="sponsored" target="_blank">bitwarden.com/twit</a></li> <li><a href="http://XBOW.com" rel="sponsored" target="_blank">XBOW.com</a></li> <li><a href="http://hoxhunt.com/securitynow" rel="sponsored" target="_blank">hoxhunt.com/securitynow</a></li> <li><a href="http://threatlocker.com/twit" rel="sponsored" target="_blank">threatlocker.com/twit</a></li> </ul></p>
Key Insights
- Leo Laporte's transformation began November 24, 2025, with Opus 4.5 release, moving from calling AI 'spicy autocorrect' to running multiple AI agents simultaneously on a single 81,000-line sales system project
- Steve Gibson, age 71, refuses to use AI for code generation due to preference for learning APIs directly, though uses Claude for advisory purposes on non-coding matters like mesh network configuration
- The Cold Card wallet vulnerability reduced cryptographic entropy from 256 bits to 32 bits for five years undetected, only discovered when a Chinese AI model (Kimi K3) analyzed its source code, leading to ~$100 million Bitcoin theft
- Modern LLMs treat all input as 'token soup' with no hard security boundaries between system prompts, user input, tool use, previous responses, and fetched web content, making role tags purely decorative
- Content phrasing between role tags influences model behavior more than the tags themselves, allowing role confusion attacks by formatting malicious prompts to appear as the model's own previous reasoning
- Current AI safeguards (refusals, classifiers, jailbreak resistance) are ineffective because they don't remove knowledge from underlying model weights, only restrict output generation
- GRAM (Gradient Routed Auxiliary Modules) enables partitioning restricted knowledge into removable neural slices, allowing different model versions for different use cases without prohibitive retraining costs
- AI cannot distinguish defensive from offensive uses of the same knowledge—cybersecurity and exploit knowledge require identical understanding, making selective knowledge restriction fundamentally difficult
- Leo uses three AI agents with different voices communicating through Buzz (a Nostr-based AI communication protocol) that verify authenticity with public-private keys to prevent spoofing between agents
- Context window limitations force large codebases to be split into 30-40 modular chunks because modern models cannot holistically understand systems exceeding their token context window
- Andy Olson created 17 applications via Claude without professional programming experience, including financial tracking, chore management, and bowling league systems, demonstrating AI democratization of development
- Richard Campbell spent $44 on Anthropic Claude in one month specifically to overcome multi-month struggle with Windows App SDK multi-tab implementation, finding AI cost-effective for breaking developmental bottlenecks
- Paul Thurrott's Notepad implementation progressed from multi-month struggle to completion through AI assistance, suggesting professional developers can achieve gains equal to amateurs but from different baselines
- Temperature-injected randomness is intentional model design for probabilistic responses rather than deterministic computation, making AI fundamentally unreliable compared to traditional software
- Leo implemented 2% improvement diminishing-returns threshold for iterative AI code refinement to prevent infinite loops, requiring explicit stopping criteria that models won't self-impose
Topics
Transcript
It's time for Security Now, and this is a very special episode. You're going to be glad you're here. We are at the Black Hat Conference in Las Vegas. Steve Gibson's here with questions from you, the audience. We're going to primarily talk about AI and security, and we've got some very special guests joining us. Security Now is next. Podcasts you love from people you trust. This is Twitter. This is Security Now with Steve Gibson. Episode 1090 recorded Wednesday, August 5th, 2026. Black hats. It's time for security now. The show we cover your security, your privacy, and how things work online. Hello, everybody. Yes, it's a little bit different. A little bit noisier because we... And Steve…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Security Now (Audio)
SN 1086: The Apex Agentic Adversary - Visual Prompt Injection Strikes
Security Now episode 1086 covers Fable 5's degraded performance due to strict safety guardrails, Chrome 150's massive 433 security fixes, visual prompt injection attacks (Inkjet), and severe FATFS library vulnerabilities affecting millions of embedded devices. The episode explores how AI is transforming both offensive and defensive cybersecurity capabilities.
SN 1085: A SOTA State-Sponsored Campaign - AI's New Superpower: Loop Engineering
Security Now Episode 1085 covers Windows 10 receiving another year of extended support, Meta's employee surveillance program backfiring with exposed data, state-sponsored credential attacks on Fortinet devices affecting 86,000+ organizations globally, and AI's emerging capability to discover vulnerabilities at scale through iteration and looping techniques.
SN 1081: AI Captured the Flag - Personal AI: Productivity Superpower or Privacy Threat?
The podcast discusses the significant impact of AI on cybersecurity practices and Capture the Flag (CTF) competitions, noting a shift in the ability to solve challenges using AI tools. This transition raises concerns about the future of skill measurement in the cybersecurity field, as traditional CTF competitions are being undermined by AI's capabilities.
SN 1076: FAST16.SYS - Unmasking the NSA's Most Diabolical Digital Sabotage
Security Now episode 1076 covers the discovery of FAST16.SYS, a sophisticated NSA-linked kernel rootkit from 2005 that subtly corrupted physics and engineering calculation software — predating Stuxnet by five years. The episode also covers a Bitwarden CLI supply chain attack, Iranian router malfunctions before the US/Israeli strikes, Meta's employee activity logging for AI training, and Steve Gibson's GRC e-commerce system rewrite.
SN 1075: Yes. Exactly. - The Zero-Day Ticking Clock
Security Now episode 1075 discusses the growing threat of AI-powered vulnerability discovery, particularly Anthropic's Project Mythos, which Mozilla confirmed found 271 bugs in Firefox. The episode also covers a disgruntled researcher publishing Windows zero-days, Microsoft's record bug bounty payouts, and a formal industry-wide warning signed by top cybersecurity leaders urging immediate action.