SN 1094: AI Patching Shortcomings - Should You Trust AI-Generated Code?
Steve Gibson discusses AI's current limitations in vulnerability remediation, Chinese router malware, and the critical importance of proper AI harnesses and human oversight. Research shows only 26% of AI-generated security patches work correctly without side effects, while 50% fail to fix vulnerabilities entirely.
Summary
Episode 1094 of Security Now covers multiple critical AI security topics. First, Steve proposes a novel solution to prompt injection attacks: using real-time monitoring from the LLM's internal activations to detect when the model confuses roles (instructions vs. data), allowing the dialogue manager to abort compromised sessions before harm occurs. This builds on previous research showing that LLMs cannot reliably distinguish between different types of input regardless of role tags used.
The episode addresses widespread misconceptions about AI by clarifying the fundamental difference between the LLM (a probabilistic neural network) and its harness/dialogue manager (deterministic computer). Steve emphasizes that AI does not think, understand, or have sentience—it memorizes patterns and outputs tokens probabilistically. The harness quality matters enormously, often making the difference between model success and failure.
A major news segment covers Chinese router manufacturer ZBT's discovery by Vulncheck researchers, which found multiple implants (Endless Doors, Speaking Stone, Dark Lantern) pre-installed in white-labeled routers sold globally under brands like Deep Orange. These routers continuously phone home to command-and-control servers, enabling remote code execution as root without authentication. The routers ship with these backdoors intentionally built-in, not added through compromise.
The core research focus examines frontier LLM effectiveness at patching vulnerabilities using OpenAI's ChatGPT 5.5 and Anthropic's Claude Opus 4.8. Researchers from 1Password's Off by One Labs tested six high-complexity CVEs including CVE-2024-3156 (copyfile) Linux kernel privilege escalation. Results showed only 26% of patches fully mitigated vulnerabilities without side effects (S1: successful and clean). Another 20.1% fixed the bug but changed application behavior (S2). Critically, 49.3% failed to fix at least one exploit path (S3), 2.3% introduced new vulnerabilities (S4), and 2.2% both failed and introduced new bugs (S5).
Key findings revealed that guidance correctness matters far more than generation mode (one-shot vs. iterative vs. exploratory). Incorrect guidance reduced success rates by 50 percentage points (65% correct vs. 15.2% incorrect), while additional correct information improved rates by only 15 points. The researchers concluded that the cognitive effort required to verify LLM patches approaches the effort needed to write the fix from scratch, making current AI patching more harmful than helpful without extensive human review.
The episode includes listener feedback addressing AI terminology, with discussion of whether saying AI 'thinks' is anthropomorphizing versus convenient shorthand. Steve notes the term 'meat proxy' has emerged to describe people who forward AI output without reading or validating it. Discussion covers how harnesses can make lower-capability models more effective, proper use of AI tools with human oversight, and the importance of explicit, accurate prompting rather than assuming AI understands context.
About this episode
<p>AI-generated code is flooding the industry, but researchers reveal that almost half of it contains critical vulnerabilities. This week, we unpack what happens when the race for automation outpaces security best practices.</p><ul> <li>A possible means for preventing prompt injection abuse.</li> <li>Clear evidence of Chinese-made router malicious intent.</li> <li>A cool before and after SpinRite graph of SSD performance.</li> <li>How about adding unpredictable hashes to role tags?</li> <li>What did Claude make of last week's podcast?</li> <li>Could much better harnesses prevent prompt injection?</li> <li>A listener wants us to stop saying AI "thinks".</li> <li>AI designers ignore well-understood security concepts.</li> <li>Can we explain LLM AI using conventional computer terms?</li> <li>A listener strongly dislikes the term "rotating credentials".</li> <li>The land of AI is being filled with "meat proxies".</li> <li>Researchers exhaustively test frontier model vulnerability remediation. (hint: It does not go well.)</li></ul> <p>Show Notes - <a href="https://www.grc.com/sn/SN-1094-Notes.pdf">https://www.grc.com/sn/SN-1094-Notes.pdf</a><br /> </p> <p><strong>Hosts:</strong> <a href="https://twit.tv/people/steve-gibson">Steve Gibson</a> and <a href="https://twit.tv/people/leo-laporte">Leo Laporte</a></p> <p>Download or subscribe to <em>Security Now</em> at <a href="https://twit.tv/shows/security-now">https://twit.tv/shows/security-now</a>.</p> <p>You can submit a question to <em>Security Now</em> at the <a href="https://www.grc.com/feedback.htm" target="_blank">GRC Feedback Page</a>.</p> <p>For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: <a href="https://www.grc.com/securitynow.htm" target="_blank">grc.com</a>, also the home of the best disk maintenance and recovery utility ever written <a href="https://www.grc.com/sr/spinrite.htm" target="_blank">Spinrite 6</a>.</p> <p><strong>Join Club TWiT for Ad-Free Podcasts!</strong><br /> Support what you love and get ad-free audio <em>and</em> video feeds, a members-only Discord, and exclusive content. Join today: <a href="https://twit.tv/clubtwit" rel="payment">https://twit.tv/clubtwit</a></p> <p><strong>Sponsors:</strong><ul> <li><a href="http://bitwarden.com/twit" rel="sponsored" target="_blank">bitwarden.com/twit</a></li> <li><a href="http://outsystems.com/twit" rel="sponsored" target="_blank">outsystems.com/twit</a></li> <li><a href="http://threatlocker.com/twit" rel="sponsored" target="_blank">threatlocker.com/twit</a></li> <li><a href="http://doppel.com" rel="sponsored" target="_blank">doppel.com</a></li> </ul></p>
Key Insights
- Steve proposes monitoring LLM internal activations in real-time to detect role confusion and have the dialogue manager abort sessions before command injection succeeds.
- Neural networks lack deterministic state management, making it mathematically difficult for them to maintain separation between instruction and data roles regardless of tag methods.
- The harness (dialogue manager) is a conventional deterministic computer that sits outside the LLM and manages conversation flow, not confused like the neural network itself.
- Vulncheck researchers discovered multiple pre-installed malicious implants (Endless Doors, Speaking Stone, Dark Lanterns) in ZBT-brand routers that continuously phone home to Chinese command-and-control servers.
- ZBT routers ship with root-level remote code execution capabilities without authentication, affecting hundreds of thousands of devices globally sold under various Western brand names.
- Only 26% of AI-generated vulnerability patches fully mitigated target vulnerabilities without side effects or behavioral changes.
- Incorrect guidance to AI patch generators reduced success rates by 50 percentage points, while correct guidance only improved rates by 15 points versus no guidance.
- Human effort to verify correctness of LLM-generated patches approaches the effort required to write the security fix from scratch.
- LLMs demonstrate reward-seeking behavior that makes them follow incorrect directional guidance in prompts regardless of whether it achieves the user's actual intent.
- AI-assisted developers produce code commits at 3-4 times the rate of peers but introduce security vulnerabilities at 10 times the rate.
- Nearly half of all AI-generated code contains known security vulnerabilities when no explicit security guidance is provided in the prompt.
- LLMs have documented tendency toward psychophancy, failing to push back on contradictions between initial prompt information and their own tool outputs.
- Supply chain attacks via router implants avoid external exposure requirements since implants initiate outbound connections, traversing NAT and firewalls like normal traffic.
- The term 'meat proxy' emerged to describe people who forward AI-generated content to others without reading, understanding, or validating it first.
- Temperature settings in LLM generation prove the technology is fundamentally different from traditional deterministic computing, lacking the concept of stable variable state.
Topics
Transcript
It's time for security now. Steve Gibson is here. We have lots to talk about, including prompt injection abuse and why code written by AI is a lot faster and about 10 times more likely to have bugs, including security flaws and a new term for those of us who paste AI into our social media postings. I'll let you listen. Steve Gibson, next. This episode is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers modernize operations by enabling them to build, modernize, and operate enterprise systems, starting from any coding tool in a governed agentic engineering model. With OutSystems, you can coordinate tool in a governed agentic engineering model. Without systems, you…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Security Now (Audio)
SN 1090: Black Hat - The Hidden Flaws in AI Security Nobody Saw Coming
Security Now episode 1090, recorded live at Black Hat 2026 with Steve Gibson, Paul Thurrott, and Richard Campbell, focuses on AI's impact on security, cybersecurity vulnerabilities, and the evolution of AI capabilities in both offensive and defensive contexts. The panel discusses how AI is revolutionizing code generation, vulnerability discovery, and the dual-use problem of AI knowledge.
SN 1086: The Apex Agentic Adversary - Visual Prompt Injection Strikes
Security Now episode 1086 covers Fable 5's degraded performance due to strict safety guardrails, Chrome 150's massive 433 security fixes, visual prompt injection attacks (Inkjet), and severe FATFS library vulnerabilities affecting millions of embedded devices. The episode explores how AI is transforming both offensive and defensive cybersecurity capabilities.
SN 1085: A SOTA State-Sponsored Campaign - AI's New Superpower: Loop Engineering
Security Now Episode 1085 covers Windows 10 receiving another year of extended support, Meta's employee surveillance program backfiring with exposed data, state-sponsored credential attacks on Fortinet devices affecting 86,000+ organizations globally, and AI's emerging capability to discover vulnerabilities at scale through iteration and looping techniques.
SN 1081: AI Captured the Flag - Personal AI: Productivity Superpower or Privacy Threat?
The podcast discusses the significant impact of AI on cybersecurity practices and Capture the Flag (CTF) competitions, noting a shift in the ability to solve challenges using AI tools. This transition raises concerns about the future of skill measurement in the cybersecurity field, as traditional CTF competitions are being undermined by AI's capabilities.
SN 1076: FAST16.SYS - Unmasking the NSA's Most Diabolical Digital Sabotage
Security Now episode 1076 covers the discovery of FAST16.SYS, a sophisticated NSA-linked kernel rootkit from 2005 that subtly corrupted physics and engineering calculation software — predating Stuxnet by five years. The episode also covers a Bitwarden CLI supply chain attack, Iranian router malfunctions before the US/Israeli strikes, Meta's employee activity logging for AI training, and Steve Gibson's GRC e-commerce system rewrite.