How to build a custom AI harness with Claude SDK
A custom AI harness is code wrapped around an AI agent to make it more effective for specific workflows. The speaker demonstrates building a Sentry bug-debugging harness using Claude SDK with a terminal UI, showing how structured constraints, custom prompts, and specific tool adapters enable agents to handle complex tasks more efficiently than general-purpose AI tools.
Summary
The episode demystifies the concept of an AI harness, explaining it as simply code around an AI agent designed to make it more effective for specific use cases. A harness consists of three core components: specific context, the ability to take specific actions, and goals for specific outcomes. The speaker built a practical example: a Sentry bug-debugging harness that investigates production issues, identifies root causes, and generates structured reports.
The harness is appropriate when workflows are repetitive, require specific setup, and have consistent outcomes—such as fixing bugs, managing production incidents, handling support escalations, or other structured processes. The speaker chose Sentry debugging because it involves coding, requires custom context, and has specific desired outcomes like creating Linear tickets and writing follow-up documentation.
The implementation uses Claude SDK as the core agent framework, connected to real tools including Sentry, Vercel, Linear, and GitHub. Rather than using general-purpose tools, the harness includes opinionated adapters that make specific API calls tailored to bug investigation rather than exposing all functionality. The interface is a terminal UI built with the Ink library, allowing intuitive navigation through investigation runs, evidence gathering, and artifacts.
Key architectural features include: custom prompts that encode the specific workflow rather than relying on natural language requests each time, tool policies that restrict what the agent can execute, structured artifact outputs that document findings in a consistent format, and the ability to run via both interactive UI and command-line tools. The speaker emphasized that building the harness required very specific prompting when using general models like GPT and Opus, as they initially resisted incorporating AI and wanted to build purely deterministic systems.
The artifact store saves evidence from each run to the file system, enabling the agent to learn from previous investigations. Output includes task runs with all messages, investigation briefs, relevant logs, worker actions, and summaries—plus formatted HTML reports showing what happened during investigation. The speaker demonstrates a real run investigating dropped edit operations, where the harness identified the issue, ranked potential root causes, and recommended whether to create a Linear issue or attempt a fix.
Key Insights
- A harness is simply code around an AI agent that makes it more effective, and the mystery around the term has been overblown—it's fundamentally about writing code to wrap AI calls to make them more efficient at specific jobs rather than open-ended tasks
- Building a custom harness for specific workflows produces better results than using general-purpose AI tools because the harness removes the need to re-explain intent and allows encoding of precise step-by-step flows that repeat consistently over time
- When building a harness with general AI models like GPT and Opus, they initially resisted incorporating AI and wanted to build purely deterministic systems, requiring very specific prompting to achieve the desired agentic behavior
- Opinionated tool adapters are more effective than exposing general MCPs—rather than letting the agent wander through all available options, the speaker built specific connectors that precisely define what data to pull from APIs in the context of bug investigation
- The speaker realized that by constraining AI work through specific harnesses and workflows, agents can solve very specific problems efficiently, changing the paradigm from open chat fields to structured, orchestrated agent work with custom outcomes
Topics
Transcript
[0:00] A harness is some code around an AI agent that makes it more effective. Why we've seen people build these specific use case harnesses is sometimes with a specific job, you just want to micromanage a little bit. You just want to be more prescriptive about how that job gets done. [music] I'm going to show you how it works and then we will talk about how I built it. So the interface I built for my harness is a terminal UI. The harness core is run on clawed agent [0:31] SDK and then it's connected to real tools. So it's connected [music] to Sentry Vcel and then it's connected to linear and GitHub in terms of getting [music]…
Full transcript available for MurmurCast members
Sign Up to AccessMore from How I AI
Stop prompting your AI agents. Start managing them.
The speaker discusses the shift from traditional agent prompting to agent management, highlighting the limitations of local Kanban board approaches and advocating for cloud-based VPS solutions with multiple communication channels to effectively manage autonomous AI agents across projects.
How I run autonomous coding agents from my phone with OpenAI Symphony + Linear
Alessio Finelli demonstrates how he uses OpenAI's Symphony framework combined with Linear and Codex to run autonomous coding agents from his phone, managing both software engineering tasks and a Pokémon card trading business through cloud-based VPS infrastructure instead of local machines.
My taste and the automated benchmark disagreed almost completely
The speaker discusses discrepancies between their subjective evaluation of model performance and automated benchmark results, noting that different judges (Opus, 4A, 5.5) show varying levels of generosity and bias. They conclude that personal judgment matters significantly and plan to incorporate more subjective taste into evaluation metrics while retiring saturated benchmark tasks.
Task-by-task model recommendations
The speaker provides task-specific model recommendations across different use cases, suggesting GPT 5.5 for PRDs, Sonnet 4.6 for prototyping and casual interaction, and Opus 4.8 or Sonnet 5 for codebase work. Model selection varies based on complexity, with Opus 4.8 excelling at dense UI design and Sonnet suitable for simpler implementations.
The How I AI Bench
The speaker introduces 'The How I AI Bench,' a new set of human and AI-graded benchmarks designed to evaluate language models on practical tasks like writing PRDs, solving bugs, and designing systems. They test Claude Sonnet 3.5 against these benchmarks and note that while it scores lower than some specialized benchmarks (69% on Agentic Coding SweetBench Pro, 82% on Terminal Bench 2.1), the difference may not be noticeable in real-world usage.