Why Medical AI Needs a Referee | Protege's Engy Ziedan

The podcast features Engy Ziedan discussing why medical AI requires robust, independent evaluation mechanisms. Ziedan explains that Protégé was founded on the insight that frontier models are limited by available training data, and the company pivoted from serving startups to working with large foundation models and enterprise customers seeking data for pre-training, mid-training, and now crucially, for benchmarking and evaluation.

A central problem Ziedan identifies is the gap between benchmark performance and real-world clinical utility. While models can score 92% on medical licensing exams, they perform at only 45% on actual clinical tasks. This disconnect occurs because traditional benchmarks rely on multiple-choice medical questions that models may have memorized rather than truly reasoning through. Additionally, benchmarks don't capture the real-world context of clinical decision-making, physician preferences, and the subtle misalignments that are harder to detect than catastrophic failures.

Ziedan argues that evals are economically critical for AI market equilibrium. Unlike Uber or past technologies where value became obvious through user adoption, AI pricing requires understanding what's actually valuable versus valueless. Without accurate evaluations, healthcare systems cannot make informed purchasing decisions, and misaligned AI systems could cause systemic harm similar to the opioid epidemic—except that with AI's speed and agency, problems escalate faster.

The transcript discusses how every AI vendor claims to be best, but without independent evaluation, no one actually knows which models perform better for specific tasks. Ziedan notes that recent competing papers in Nature and arXiv reached opposite conclusions about whether general frontier models or vertical-specific healthcare AI performs better, demonstrating the contamination and methodology problems plaguing current evaluations.

Ziedan also addresses the subtler problem of misalignment beyond catastrophic failures. He gives the example of AI systems used in insurance prior authorization and medical denials that may be optimizing for revenue minimization rather than patient outcomes, creating situations where AI might check a patient's debt before recommending treatment. This type of misalignment is harder to define and detect than outright failure but represents a profound problem in healthcare.

Protégé's solution involves hosting unified evaluations within specific healthcare subnodes (like oncology pathology or clinical trial recommendations), producing comparative reports where multiple vertical AI builders can be fairly evaluated against each other. The methodology remains transparent, competitor identities are concealed until the end, and they've implemented a strict data membrane: no patient data that entered any model's training can be used in benchmarking, requiring them to source entirely new pathology slides and clinical data.

Ziedan argues Protégé is well-positioned for this referee role because they have comparative advantage in healthcare data and can identify precisely which data would improve a model's performance on revealed deficiencies. Crucially, their incentive structure favors impartiality—any loss of trust destroys the evaluation network immediately. He contrasts this with government oversight, which he argues is neither feasible nor fast enough given AI's rapid evolution, and notes the government itself has asked industry collaborators for help identifying evaluation standards.