My taste and the automated benchmark disagreed almost completely

The speaker describes an evaluation process where multiple language models served as judges to assess other models' outputs. They tested models including Opus, Claude 4A, and GPT 5.5, with particular focus on whether judges showed inherent bias toward themselves. A key finding was that GPT 5.5 emerged as the toughest judge despite being evaluated, and the speaker expresses a preference for using it as a judge. However, 5.5 judged itself more harshly than other judges evaluated it, suggesting some self-critical bias. Overall, the judges showed general agreement but tended toward generosity in their assessments. To balance these tendencies, the speaker implemented a double-bench approach with multiple judges. The speaker reflects that their personal taste diverged significantly from what the metrics indicated, leading to the conclusion that subjective evaluation ('vibe checks') shouldn't be dismissed. As a result, they plan to encode more of their personal taste directly into the judgment framework. Additionally, they identify that certain benchmark tasks have become saturated—specifically mentioning agentic bug tracking—where all models perform comparably well, making these tasks poor discriminators for evaluation purposes.