SN 1093: Tokens in the Stream - Why LLMs are inherently insecure and prompt injection will persist
Steve Gibson discusses a research paper revealing that LLMs identify roles (user, system, tool, thinking) primarily through writing style rather than tags, making them fundamentally vulnerable to prompt injection attacks. This architectural flaw stems from LLMs being statistical token prediction machines with no formal parsing or state management, and cannot be fixed without redesigning the entire system.
Summary
Steve Gibson presents an in-depth analysis of a research paper titled 'Prompt Injection as Role Confusion,' examining why large language models are inherently insecure and vulnerable to prompt injection attacks. The episode begins with discussions of AI model distillation—how frontier AI companies train newer models using outputs from existing mature models—and news about Anthropic expanding access to Claude Mythos 5 through backend integration with cybersecurity tools rather than direct user access.
Gibson explains that modern LLMs process all information as a continuous stream of tokens partitioned into labeled segments using tags: system (provider instructions), user (human input), assistant (model output), tool (external data), and thinking (internal reasoning). However, the fundamental problem is that LLMs don't actually parse these tags in a formal, state-based way. Instead, the research demonstrates that models infer role from the style and content of text rather than from the tags themselves.
The researchers conducted three key experiments. First, they measured 'COT-ness' (chain-of-thought-ness) to quantify how much the model treats text as internal reasoning. Second, they stripped all tags from conversations and found that tokens originally in think tags still registered high COT-ness—the model didn't need the tags because the reasoning-like writing style was sufficient. Third, they wrapped all text in user tags but found that text styled like reasoning still triggered internal reasoning features, proving that style overrides explicit tags.
This creates a fundamental security architecture failure: LLMs identify roles like a person identifies professions by appearance rather than by checking ID. Attackers exploiting this inject malicious commands styled to match the higher-privilege role (user or think) within lower-privilege contexts (tool tags from untrusted web pages). By carefully wording injected text to sound like user commands or reasoning, attackers achieve 60% success rates against frontier models with near-zero baseline defenses.
Gibson emphasizes this isn't lazy engineering but an unavoidable consequence of how LLMs work. They are massive neural networks processing tokens sequentially, predicting the next most likely token based on weights set during training. There is no actual 'mode' or 'state'—no way to set a tool-reading mode that persists until a closing tag. Attempts to fix this through additional metadata bits or tag interleaving have failed. The entire system is, as Loren Kohnfelder (a legendary security researcher and 50-year friend of Gibson's) described it, a 'token soup' where everything sits in one undifferentiated stream with no hard security boundary.
The episode traces how this architecture emerged organically rather than by design. In 2020, users manually formatted prompts with 'user:' and 'assistant:' labels. In 2022, ChatGPT formalized these into system tags. New tags (tool, think) were added ad hoc to solve engineering problems, never as part of a principled security framework. This means roles went from formatting tricks to the most load-bearing infrastructure in the LLM stack, despite having no formal design.
Gibson discusses why current defenses fail: attack memorization (learning to refuse known attack phrases) is brittle against adaptive human attackers, and robust role perception is theoretically impossible given how models actually work. Recent papers show frontier models still fail 11-25% of the time against automated attacks, with much higher failure rates against skilled humans. Static benchmarks measure only attacks models have already learned to recognize, not novel adaptive attacks.
The episode also covers practical security solutions: Anthropic integrating Mythos 5 into backend services so users don't directly access the model; Claude Security scanning code with Mythos 5 while preventing access to the model itself; a $35 million 'Defender Advantage Fund' for open-source security; and Bitwarden Secrets Manager for securely scoping agent access to credentials without exposing them to agents that might be compromised by prompt injection.
About this episode
<p>Turns out, every chatbot conversation runs on a messy hack at the heart of language models, making prompt injection an unsolved—and possibly unsolvable—security threat. Steve and Leo unravel the research that explains why "roles" in AI aren't what you think they are.</p><ul> <li>Understanding the controversy surrounding "AI Model Distillation"</li> <li>Anthropic moves to make their most powerful Mythos 5 model more widely available.</li> <li>Bitwarden's "Secrets Manager" offering prevents agentic and prompt injection abuse.</li> <li>The astounding and disturbing truth about the way conversation AI actually works</li></ul> <p>Show Notes - <a href="https://www.grc.com/sn/SN-1093-Notes.pdf">https://www.grc.com/sn/SN-1093-Notes.pdf</a></p> <p><strong>Hosts:</strong> <a href="https://twit.tv/people/steve-gibson">Steve Gibson</a> and <a href="https://twit.tv/people/leo-laporte">Leo Laporte</a></p> <p>Download or subscribe to <em>Security Now</em> at <a href="https://twit.tv/shows/security-now">https://twit.tv/shows/security-now</a>.</p> <p>You can submit a question to <em>Security Now</em> at the <a href="https://www.grc.com/feedback.htm" target="_blank">GRC Feedback Page</a>.</p> <p>For 16kbps versions, transcripts, and notes (including fixes), visit Steve's site: <a href="https://www.grc.com/securitynow.htm" target="_blank">grc.com</a>, also the home of the best disk maintenance and recovery utility ever written <a href="https://www.grc.com/sr/spinrite.htm" target="_blank">Spinrite 6</a>.</p> <p><strong>Join Club TWiT for Ad-Free Podcasts!</strong><br /> Support what you love and get ad-free audio <em>and</em> video feeds, a members-only Discord, and exclusive content. Join today: <a href="https://twit.tv/clubtwit" rel="payment">https://twit.tv/clubtwit</a></p> <p><strong>Sponsors:</strong><ul> <li><a href="http://threatlocker.com/twit" rel="sponsored" target="_blank">threatlocker.com/twit</a></li> <li><a href="https://www.box.com/ai?utm_source=youtube&utm_medium=paidinfluencer&utm_theme=icm&utm_campaign=FY27_Q1_TWiT" rel="sponsored" target="_blank">box.com/AI</a></li> <li><a href="http://hoxhunt.com/securitynow" rel="sponsored" target="_blank">hoxhunt.com/securitynow</a></li> <li><a href="http://guardsquare.com" rel="sponsored" target="_blank">guardsquare.com</a></li> <li><a href="https://material.security" rel="sponsored" target="_blank">material.security</a></li> </ul></p>
Key Insights
- LLMs process all information as a single continuous token stream with no formal mode states or parsing, making it impossible to create hard security boundaries between different types of input regardless of tags used.
- Models infer the role of text primarily from writing style and content similarity rather than from explicit XML-like tags, meaning a command styled like user input will be treated as user input even if tagged as tool output.
- Removing all tags from LLM input results in virtually unchanged model behavior for reasoning text, proving that formatting punctuation is not the security mechanism—the style of the writing itself is.
- Prompt injection succeeds when attackers embed commands in untrusted contexts (like tool outputs from web pages) but phrase them to sound like user commands, exploiting the fact that sound-alike writing matters more to the model than tag labels.
- The 'token soup' problem is not a fixable engineering oversight but a fundamental architectural constraint of how neural networks process sequential text without state variables or mode flags.
- Frontier models showed 60% attack success rates against chain-of-thought forgery attacks that inject fabricated reasoning into prompts, with only 10% success when styling was removed, proving style is the vulnerability.
- Role tags evolved as ad hoc formatting tricks starting in 2020-2022 to solve immediate engineering problems, not as part of any designed security framework, making them poorly suited for security purposes.
- Attempts to fix role confusion through metadata bits, tag interleaving, or other technical modifications have all failed to meaningfully improve security without unacceptable performance costs.
- Attack memorization (training models to refuse known attack phrases) is brittle because skilled human attackers can rephrase and adapt attacks until they find variants the model hasn't learned to refuse.
- Recent papers from May 2026 found that GPT-5.4 and Opus 4.5 still fail against automated prompt injection attacks 25% and 11% of the time respectively, with much higher failure rates against adaptive human attackers.
- Anthropic's strategy to safely expose Mythos 5 involves running it on backend infrastructure where users only receive specific filtered outputs (like patches or findings) rather than direct model access.
- The Bitwarden Secrets Manager solves the problem of AI agents needing credentials to act on behalf of humans by scoping access so agents only access the specific secrets needed for their task.
- LLMs identify roles like identifying a stranger's profession from how they talk and dress rather than checking their ID, making them inherently vulnerable to adversaries who can mimic higher-privilege styles.
- Every role boundary (think/assistant, user/assistant, user/tool) was created to isolate competing objectives so they could be optimized independently, but when role confusion occurs, these objectives bleed into each other.
- The phrase 'role tags were a formatting trick that became the security architecture' encapsulates the core problem: security was built on something never designed for it, making the current system fundamentally unfixable without complete redesign.
Topics
Transcript
It's time for Security Now. Steve Gibson is here. This is going to be another banger of an episode. Steve has found a paper that talks about the underlying mechanism in an LLM and why it is not only inherently insecure, it will never be anything but insecure. How AIs think. Coming up next on Security Now. Podcasts you love. From people you trust. This is Twit. This is Security Now with Steve Gibson. Episode 1093. Recordeduesday august 25th 2026 tokens in the stream it's time for security now yay tuesday has come around and that means so as this guy right here mr steve gibson he's here to regale us with tales of cyber security hello steve oh leo…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Security Now (Audio)
SN 1094: AI Patching Shortcomings - Should You Trust AI-Generated Code?
Steve Gibson discusses AI's current limitations in vulnerability remediation, Chinese router malware, and the critical importance of proper AI harnesses and human oversight. Research shows only 26% of AI-generated security patches work correctly without side effects, while 50% fail to fix vulnerabilities entirely.
SN 1090: Black Hat - The Hidden Flaws in AI Security Nobody Saw Coming
Security Now episode 1090, recorded live at Black Hat 2026 with Steve Gibson, Paul Thurrott, and Richard Campbell, focuses on AI's impact on security, cybersecurity vulnerabilities, and the evolution of AI capabilities in both offensive and defensive contexts. The panel discusses how AI is revolutionizing code generation, vulnerability discovery, and the dual-use problem of AI knowledge.
SN 1086: The Apex Agentic Adversary - Visual Prompt Injection Strikes
Security Now episode 1086 covers Fable 5's degraded performance due to strict safety guardrails, Chrome 150's massive 433 security fixes, visual prompt injection attacks (Inkjet), and severe FATFS library vulnerabilities affecting millions of embedded devices. The episode explores how AI is transforming both offensive and defensive cybersecurity capabilities.
SN 1085: A SOTA State-Sponsored Campaign - AI's New Superpower: Loop Engineering
Security Now Episode 1085 covers Windows 10 receiving another year of extended support, Meta's employee surveillance program backfiring with exposed data, state-sponsored credential attacks on Fortinet devices affecting 86,000+ organizations globally, and AI's emerging capability to discover vulnerabilities at scale through iteration and looping techniques.
SN 1081: AI Captured the Flag - Personal AI: Productivity Superpower or Privacy Threat?
The podcast discusses the significant impact of AI on cybersecurity practices and Capture the Flag (CTF) competitions, noting a shift in the ability to solve challenges using AI tools. This transition raises concerns about the future of skill measurement in the cybersecurity field, as traditional CTF competitions are being undermined by AI's capabilities.