Sonnet 5 review: I ran 64 generations to find out if it's worth it
The creator develops the 'How I AI Bench,' a custom evaluation framework combining human vibe checks with LLM judging to assess AI models on practical tasks like PRD writing, prototyping, and agentic coding. Testing five frontier models including Claude Sonnet 5, the results reveal significant disagreements between automated scoring and human preference, with Sonnet 4.6 and Gemini 3 Pro ranking highest despite Sonnet 5 being the newest release.
Summary
The episode opens with an announcement of Claude Sonnet 5, positioned by Anthropic as offering near-Opus performance at lower costs. Rather than conducting subjective vibe checks, the creator decides to build a repeatable benchmark system called the How I AI Bench to evaluate models systematically.
Using Claude Code, the creator develops a comprehensive evaluation framework with specific design principles: frozen inputs, blind scoring where possible, and structured rubrics. The benchmark focuses on four key task categories: PRD writing (converting messy notes into product requirements), prototype generation (UI/UX design), agentic multi-step coding tasks, and agent personality/voice assessment.
The creator personally runs 64 generations across five models (identified as Gemini 3 Pro, GPT 5.5, Claude Sonnet 4.6, Claude Sonnet 5, and Claude Opus 4.8) on these tasks. For evaluation, they use a dual-scoring method: personal 1-5 vibe checks based on gut instinct and shipping readiness, plus automated LLM judging from both GPT 5.5 and Opus 4.8 to assess for objective issues like broken code and constraint violations.
The initial leaderboard surprises the creator: Gemini 3 Pro and Sonnet 5 tie at the top, with GPT 5.5 close behind, while Opus 4.8 and Sonnet 4.6 rank poorly. However, the creator's manual scores directly contradict the automated results—they actually prefer Sonnet 4.6 as best and rank Gemini 3 Pro worst. This major discrepancy reveals that LLM judges tend to cluster toward middle-of-the-curve scores and miss subjective qualities like 'taste' and 'voice.'
Analyzing the disagreement, the creator identifies their bias against 'Claude slop' (characteristic Claude writing patterns) and notes that automated judges flagged functional issues in prototypes the human reviewer hadn't assessed. The creator realizes they evaluated wireframes purely visually without checking functionality.
Ultimately, the creator produces a weighted index (70% human judgment, 30% automated scoring) and provides task-specific model recommendations: GPT 5.5 for PRDs, Sonnet 4.6 for prototypes and conversation, Opus 4.8 for complex design work, and Sonnet 5 for codebase analysis.
About this episode
<p>I’ve been testing every major frontier model release since the start of the year, and when Anthropic dropped Sonnet 5, I wanted more than a vibe check. I got tired of one-off tests I couldn’t repeat or compare over time, so I built something better: the How I AI Bench, a repeatable eval harness I constructed live using Claude Code while recording this episode. I ran Sonnet 5 blind against four other frontier models (Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro) across PRD quality, prototype generation, agentic task completion, and agent personality. The results were not what I expected.</p><p><br /></p><p><strong>What you’ll learn:</strong></p><ol><li>What Anthropic claims Sonnet 5 improves over Sonnet 4.6, and where the benchmark data actually backs that up</li><li>How I built the How I AI Bench in under 45 minutes using Claude Code, starting from my own stored session history</li><li>Why I combined human vibe scoring (70%) with LLM as judge scoring (30%) instead of trusting either alone</li><li>How to set up a local HTML scoring page so you can rate AI outputs on gut feel and export those scores as JSON</li><li>Which model I recommend for PRDs, which for complex prototypes, and which for chatting with an agent daily</li></ol><p>—</p><p><strong>Brought to you by:</strong></p><p><a href="https://runwayml.com/howIAI"><strong>Runway</strong></a>—The creative AI platform for images, video and more</p><p><a href="https://hyperagent.com/howiai"><strong>Hyperagent</strong></a>—Deploy fleets of agents that handle real work</p><p>—</p><p><strong>In this episode, we cover:</strong></p><p>(00:00) Sonnet 5 is out</p><p>(01:55) What Anthropic claims</p><p>(04:02) Why I’m done with one-off vibe checks</p><p>(05:05) Building the How I AI Bench live with Claude Code</p><p>(07:42) The scoring system</p><p>(10:43) Agent voice eval</p><p>(11:57) Quick recap</p><p>(13:58) Results: The How I AI index leaderboard</p><p>(21:21) What I’m improving for the next run</p><p>(22:16) Generating a Claire-weighted index</p><p>(23:53) Model-by-task recommendations</p><p>—</p><p><strong>Tools referenced:</strong></p><p>• Claude Sonnet 5: <a href="https://www.anthropic.com/news/claude-sonnet-5">https://www.anthropic.com/news/claude-sonnet-5</a></p><p>• Claude Opus 4.8: <a href="https://www.anthropic.com/news/claude-opus-4-8">https://www.anthropic.com/news/claude-opus-4-8</a></p><p>• GPT-5.5 (OpenAI): <a href="https://openai.com/index/introducing-gpt-5-5/">https://openai.com/index/introducing-gpt-5-5/</a></p><p>• Gemini 3 Pro (Google DeepMind): <a href="https://deepmind.google/models/gemini/pro/">https://deepmind.google/models/gemini/pro/</a></p><p>• Cursor: <a href="https://www.cursor.com/">https://www.cursor.com/</a></p><p>—</p><p><strong>Other references:</strong></p><p>• SWE-bench Pro (agentic coding benchmark referenced): <a href="https://www.swebench.com/">https://www.swebench.com/</a></p><p>—</p><p><strong>Where to find Claire Vo:</strong></p><p>ChatPRD: <a href="https://www.chatprd.ai/">https://www.chatprd.ai/</a></p><p>Website: <a href="https://clairevo.com/">https://clairevo.com/</a></p><p>LinkedIn: <a href="https://www.linkedin.com/in/clairevo/">https://www.linkedin.com/in/clairevo/</a></p><p>X: <a href="https://x.com/clairevo">https://x.com/clairevo</a></p><p>—</p><p>Production and marketing by <a href="https://penname.co/">https://penname.co/</a>. For inquiries about sponsoring the podcast, email [email protected].</p>
Key Insights
- The creator discovered that LLM-based judges consistently rate outputs toward the middle of the distribution and lack the subjective judgment to recognize qualities like 'taste' or distinctive personality, despite these being important to humans.
- Personal human preference for Claude Sonnet 4.6 directly contradicted the automated benchmark results that ranked it poorly, suggesting that current automated evaluation methods fail to capture what actual users value in model behavior.
- The creator recognized they evaluated prototypes on visual appearance alone without assessing functionality, revealing a gap in their methodology that automated judges caught but manual assessment initially missed.
- Anthropic's claim that Sonnet 5 offers 'Opus-level performance at Sonnet-level prices' was not supported by this blind testing, with Sonnet 5 ranking at the bottom of the creator's weighted preference list despite being the newest model.
- Building repeatable, task-specific benchmarks with frozen inputs revealed that model performance varies significantly by task type, with no single model dominating across PRD writing, prototyping, coding, and conversational ability simultaneously.
Topics
Transcript
We've got a new model, people, and it's from Anthropic. Now, is it Mythos? No. Is it Fable? No. But it is Claude Sonnet 5. Anthropic is claiming it's the most agentic Sonnet model yet, and we will get Opus-level tasks at Sonnet-level prices. Now, I've been testing a lot of models, and I'm starting to get bored of doing the vibe check. What I want to start developing is a set of models and I'm starting to get bored of doing the vibe check. What I want to start developing is a set of benchmarks we can regularly test these new models against that you'll care about. So today I'm going to be introducing the How I AI Bench,…
Full transcript available for MurmurCast members
Sign Up to AccessMore from How I AI
This solo builder runs 24/7 local AI on his own hardware | Alex Finn
Alex Finn, a solo builder, discusses his setup of multiple local AI machines (Mac Studios, DGX Spark, RTX 5090) running 24/7 to power autonomous workflows for software development, security scanning, and market research. He explains why local models unlock unlimited AI usage compared to cloud APIs, and demonstrates his 'software factory' with automated build and review loops using Claude Code.
GPT-5.6 Sol vs. Claude Fable: Why OpenAI’s new model crushes my benchmark
A creator conducts a detailed benchmark comparing OpenAI's new GPT-5.6 models (Sol, Terra, Luna) against Claude Fable across real-world tasks like PRD writing, prototyping, and coding. GPT-5.6 Sol emerges as the superior choice for practical product development, offering better design uniqueness, clearer communication, and more effective collaboration despite Fable's theoretical intelligence.
What a harness is and how to build one with Claude Agent SDK
A harness is custom code built around an AI agent to make it more effective for specific workflows. The speaker demonstrates building a Sentry bug-fixing harness using Claude Agent SDK, explaining how structured constraints, opinionated tooling, and custom prompts enable AI agents to solve complex tasks more reliably than general-purpose coding tools.
How I run autonomous coding agents from my phone with OpenAI Symphony + Linear | Alessio Fanelli (Kernel Labs)
Alessio Fanelli demonstrates how to run autonomous coding agents from a phone using OpenAI's Symphony framework integrated with Linear, showing practical applications from Pokemon card arbitrage to small business automation. He emphasizes moving from agent prompting to agent management through cloud-based orchestration and shares how AI enables efficient scaling of traditionally manual, inefficient business processes.