Running evals on your internal agents is how you keep them trustworthy over time
The evaluation of internal AI agents is crucial for maintaining their reliability. An internal eval platform allows engineers to review agent performance regularly and ensure their responses are accurate, particularly in critical areas like coding.
Summary
In the discussion, it is emphasized that regular evaluations of internal AI agents are essential for maintaining trustworthiness over time. Each evaluation generates a log that is handled by engineers who analyze whether the agent's responses are correct and satisfactory. This process mirrors the evaluations conducted for customer-facing AI products, highlighting the importance of maintaining high standards for AI, especially those involved with important tasks such as coding. Continuous assessment allows for ongoing improvements in the performance of these internal tools.
Key Insights
- Regular evaluations of internal agents are logged into an internal eval platform for review by engineers.
- Engineers assess whether agents get answers right or wrong as part of the evaluation process.
- The scoring mechanism of agents is subject to scrutiny during evaluations to ensure quality.
- Internal evaluations mirror those used for customer-facing AI products, emphasizing their importance.
- Special attention is needed for internal AI bots that are involved in critical tasks such as coding.
Topics
Transcript
[0:00] A lot of great folks run evals on this internal agent. So every time this review is run, it gets logged into an internal eval platform and an engineer looks at it and says, "Did the agent get this right? Did the agent get this wrong? Are we happy with the scoring mechanism?" So, very similar to how you'd use evals to improve your customerf facing AI products, you're going to want to use evals to improve your internally facing AI bots, especially ones that touch really critical things [music] like code.
Full transcript available for MurmurCast members
Sign Up to AccessMore from How I AI
You don't have to review every AI-generated PR
The speaker describes an AI agent framework that automatically reviews pull requests by scoring them across six risk factors using markdown-based instructions and skills. PRs are classified as low (0-24 points), medium (25-64), or high risk (65+), with only medium and high risk requiring human approval.
Voice is the highest-bandwidth input channel for AI
Voice interaction with AI addresses the common problem of 'blank chat window syndrome' by enabling users to naturally dump context through speech rather than carefully constructed text prompts. Voice-to-voice AI interaction mirrors the experience of delegating to a human assistant, making it a higher-bandwidth and more natural way to communicate with LLMs.
ChatGPT voice doesn't just respond to you: it operates your computer in parallel
ChatGPT's voice interface can now operate a user's computer in parallel, handling tasks like booking flights and hotels in the background while the user continues with other work. The user demonstrates asking the AI to check their calendar, find travel options, and book accommodations for a Paris offsite, with the AI managing browser automation to complete these logistical tasks autonomously.
How this OpenAI engineer uses Codex + ChatGPT Work to automate everything
Nick Bowman from OpenAI demonstrates advanced use cases of ChatGPT and Codex, including voice-orchestrated multi-threaded task management, AI-powered website creation via ChatGPT artifacts, and automated video editing for content creation. He emphasizes how these tools are becoming accessible to non-technical users through mobile and voice interfaces.
How (and why) to build a personal API
The conversation explores the concept of personal APIs that store individual preferences like favorite restaurants and coffee orders, enabling others to make thoughtful gestures without asking. The discussion evolves to envision AI agents autonomously accessing these APIs to perform personalized tasks like making reservations.