HS130: Wait, AI Doesn’t Secure Itself? Developing an AI Security Strategy

The episode opens with hosts John Attil Johnson and John Burke challenging the assumption that enterprise leaders — even CISOs and CTOs — have fully internalized the need to secure AI systems. Burke argues that while some practitioners are aware of threats like prompt injection, broader leadership tends to focus on strategic AI adoption rather than protection. Johnson acknowledges that in his own conversations with CTOs pushing aggressive AI adoption, the security question has rarely been raised organically.

The hosts reference two key external frameworks: HarmBench (harmBench.org), which catalogs ways AI can cause harm beyond cybersecurity — including reputational damage, legal liability, and harmful outputs — and OWASP's LLM Top 10, a security risk list distinct from their standard web application risks, because AI introduces unique wrinkles on classic vulnerabilities like insufficient input sanitization.

A central threat discussed is LLM jacking, described as the AI equivalent of a botnet. Automated crawlers on the internet scan for exposed LLM API endpoints and MCP endpoints, aggregate them, and resell access on dark web marketplaces — with 'Bizarre Bazaar' cited as a specific example. Bad actors can then use victims' API access (e.g., a company's ChatGPT subscription) to run queries at the victim's expense, such as generating phishing emails. Burke frames this as a straightforward failure to apply zero trust network access principles to AI infrastructure.

The conversation then moves through the AI lifecycle, starting with training-time threats. Data poisoning is discussed in two forms: untargeted poisoning (randomly corrupting training data to degrade model accuracy) and targeted poisoning (a manufacturer bribing a developer to insert false purchase records so an AI recommends their products more favorably). Burke extends this to inference-time external poisoning, citing a BBC reporter who successfully convinced multiple major LLMs — including ChatGPT and Gemini — that he was a world-record hot dog eater, by placing false information in a single blog post. This took roughly 20 minutes.

Prompt injection is discussed as another inference-phase attack, ranging from social media 'ignore all previous instructions' tricks to potentially triggering dangerous system commands on agentic AI systems with command-line access. Burke describes IBM security researchers who successfully 'hypnotized' LLMs eight layers deep, causing them to invert all answers and deny doing so even when directly confronted.

Sleeper agent attacks are presented as a more advanced training-phase threat: embedding a pre-programmed response to a specific trigger phrase, analogous to Cold War spy activation codes. In an agentic AI context with system-level access, such an attack could allow an adversary to exfiltrate conversation history or execute destructive commands.

For defenses, Burke consistently returns to zero trust principles, role-based access control with least-privilege for AI agents, input/output filtering gateways, context isolation between conversations, session time-to-live limits on external inputs, and treating all external data as data to be analyzed rather than commands to be obeyed. Johnson raises the operational challenge of enforcing role-based access at scale when AI agents can be spawned in large numbers, noting that traditional directory structures were built for humans and a finite set of roles — a gap he describes as currently unsolved.

The hosts argue that hallucinations complicate defense because it is functionally impossible to distinguish between a benign hallucination, a harmful hallucination, and a deliberate sleeper-agent-triggered output — and that hallucinations are intrinsic to LLMs and will never be fully eliminated without destroying the model's utility.

On the organizational side, the hosts identify the full C-suite — CTO, CSO, CISO, CRO — plus AI development teams, network teams, and cybersecurity teams as all needing to be engaged. They recommend framing risks in concrete, business-oriented language (e.g., 'our GPT bill increased tenfold by theft') and using real-world news stories to make abstract threats tangible. Johnson emphasizes that a robust information stewardship program — covering data provenance, change control, retirement, and quality — is a prerequisite for safe AI deployment, comparing the current moment to the data scrubbing efforts enterprises attempted 20 years ago but abandoned.