WTF is a harness?`
A harness is code that wraps around an AI agent to make it more effective. The speaker demonstrates how a harness can automatically investigate production issues by analyzing Sentry warnings, identifying root causes, determining impact, and recommending whether to create tickets or apply fixes.
Summary
The speaker explains that a harness is fundamentally a code wrapper around an AI agent designed to enhance its effectiveness and has changed their perspective on how work gets accomplished. They then walk through a concrete example of a harness in action investigating a document operations issue. The AI agent generated an investigation brief that confirmed a Sentry warning existed, determined it impacted 150 users and is still occurring hourly, and clarified that it's a warning rather than a critical error. The harness enabled the AI to identify multiple potential root causes and even pinpoint a blind spot in a specific function. The agent provided exact locations in the product where the issue manifests and offered verification steps by suggesting how to fetch raw Sentry events to validate the identified problems. Finally, the harness-enabled AI made recommendations about whether to create a Linear issue (determining yes, it should be created and assigned) and whether to enter patch mode for immediate fixes (recommending against it).
Key Insights
- A harness is code around an AI agent that makes it more effective and has fundamentally changed how work gets done
- The harness-enabled AI confirmed a Sentry warning affecting 150 users and determined it's still occurring hourly but is a warning rather than an actual error
- The AI identified not only multiple potential root causes but also a blind spot in a specific function within the codebase
- The harness provided exact product surface locations where the issue occurs and offered specific verification steps using raw Sentry events
- The AI made a determination that a Linear issue should be created and assigned but recommended against entering patch mode for an immediate fix
Topics
Transcript
[0:00] A harness is just code around an AI agent that makes it more effective. It's really like changed my mind about how work gets done. Okay, so let's see what it actually generated. Here's the investigation brief on that edit document operations dropped. I have confirmed evidence. So it's saying, "Yes, there was definitely a sentry warning. It's impacted 150 users. It's still happening hourly. It's a warning, so it's not an actual error." And then it found likely root causes. And so it's identified a couple potential root [0:31] causes as well as a blind spot in this particular function. It's told me exactly where in the product surface the issue is and then how I would actually…
Full transcript available for MurmurCast members
Sign Up to AccessMore from How I AI
Context offloading is an underrated AI use case
The speaker highlights context offloading as an underrated AI use case, where AI serves as a safety net for routine cognitive tasks like email management and personal finances. Rather than adding new capabilities, AI reduces anxiety about missing important information or making mistakes by handling monitoring tasks, thereby freeing up mental bandwidth.
Stop prompting your AI agents. Start managing them.
The speaker discusses the shift from traditional agent prompting to agent management, highlighting the limitations of local Kanban board approaches and advocating for cloud-based VPS solutions with multiple communication channels to effectively manage autonomous AI agents across projects.
How to build a custom AI harness with Claude SDK
A custom AI harness is code wrapped around an AI agent to make it more effective for specific workflows. The speaker demonstrates building a Sentry bug-debugging harness using Claude SDK with a terminal UI, showing how structured constraints, custom prompts, and specific tool adapters enable agents to handle complex tasks more efficiently than general-purpose AI tools.
How I run autonomous coding agents from my phone with OpenAI Symphony + Linear
Alessio Finelli demonstrates how he uses OpenAI's Symphony framework combined with Linear and Codex to run autonomous coding agents from his phone, managing both software engineering tasks and a Pokémon card trading business through cloud-based VPS infrastructure instead of local machines.
My taste and the automated benchmark disagreed almost completely
The speaker discusses discrepancies between their subjective evaluation of model performance and automated benchmark results, noting that different judges (Opus, 4A, 5.5) show varying levels of generosity and bias. They conclude that personal judgment matters significantly and plan to incorporate more subjective taste into evaluation metrics while retiring saturated benchmark tasks.