My taste and the automated benchmark disagreed almost completely
The speaker discusses discrepancies between their subjective evaluation of model performance and automated benchmark results, noting that different judges (Opus, 4A, 5.5) show varying levels of generosity and bias. They conclude that personal judgment matters significantly and plan to incorporate more subjective taste into evaluation metrics while retiring saturated benchmark tasks.
Summary
The speaker describes an evaluation process where multiple language models served as judges to assess other models' outputs. They tested models including Opus, Claude 4A, and GPT 5.5, with particular focus on whether judges showed inherent bias toward themselves. A key finding was that GPT 5.5 emerged as the toughest judge despite being evaluated, and the speaker expresses a preference for using it as a judge. However, 5.5 judged itself more harshly than other judges evaluated it, suggesting some self-critical bias. Overall, the judges showed general agreement but tended toward generosity in their assessments. To balance these tendencies, the speaker implemented a double-bench approach with multiple judges. The speaker reflects that their personal taste diverged significantly from what the metrics indicated, leading to the conclusion that subjective evaluation ('vibe checks') shouldn't be dismissed. As a result, they plan to encode more of their personal taste directly into the judgment framework. Additionally, they identify that certain benchmark tasks have become saturated—specifically mentioning agentic bug tracking—where all models perform comparably well, making these tasks poor discriminators for evaluation purposes.
Key Insights
- GPT 5.5 was consistently the toughest judge across evaluations, even though it judged itself lower than other judges evaluated it
- Multiple judges show overall agreement but demonstrate a systematic tendency toward generosity in their assessments
- The speaker's personal taste and subjective evaluation diverged significantly from what automated metrics indicated
- Model performance depends on the specific task and how well a model's strengths fit that particular task type
- Saturated tasks like agentic bug tracking don't function as effective benchmarks because all evaluated models perform comparably well on them
Topics
Transcript
[0:00] We had a model as a judge, and so we had Opus 4A and 5.5 judge itself. I had the benchmark check if there was any inherent bias, like did Opus like 5.5 better? I've consistently seen GPT 5.5 be the toughest judge, and so I actually prefer a 5.5 judge, but it judged itself lower than the other judge did. The judges overall agree, but they were overall generous, and sort of balancing these two judges is exactly why we ran [0:30] this double bench. Takeaways, the model's going to depend on the job and the strength of the model fit by task. I would say my taste actually matters, so maybe those vibe checks are not bad,…
Full transcript available for MurmurCast members
Sign Up to AccessMore from How I AI
Your prompt is a spec, and a good spec is the whole game
A prompt acts as a specification that defines quality criteria for AI outputs. By clearly articulating 5-10 defining characteristics of what makes something good (whether a garment, photo, or illustration), you provide the AI with the same level of detail a human creator would need to produce excellent results.
How I use ChatGPT to run my fashion business
Yana Welander demonstrates how she built an AI-native fashion brand using ChatGPT and Codex as her technical co-founder, automating design, production, and business operations from sketches to manufacturing and e-commerce. She shares how AI unlocks previously impossible garments while highlighting remaining challenges in pattern generation and realistic image rendering.
The biggest barrier to AI adoption is not fear of technology. It's the absence of muscle memory.
The speaker argues that the primary barrier to AI adoption is not fear but lack of muscle memory—the habitual use of AI tools. Success requires building collaborative practices through empathy, accessible training, and establishing regular workflows rather than overcoming technological anxiety.
Staying in Gmail means none of your AI work compounds
The speaker discusses how staying within Gmail limits the compounding value of AI-assisted work and writing, and demonstrates a workflow using Claude to enhance Gmail's functionality by adding formatting, links, and draft management capabilities.
Intent engineering beats prompt engineering every time
The speaker emphasizes the importance of integrating AI, specifically Claude, into daily workflows by creating reminders to seek its assistance. This approach allows for collaborative problem-solving, transforming static prompts into dynamic conversations.