You Can't Punish a Model Into Alignment - Ajeya Cotra
Ajeya Cotra argues against punitive approaches to AI model alignment, explaining that punishment is ineffective and counterproductive. Instead, she advocates for conducting safe counterfactual tests and third-party scientific assessments to understand model inconsistencies.
Summary
Cotra addresses a common misconception among policymakers in Washington who propose punishing AI models as a solution to alignment problems. She critiques this approach as dangerous, arguing that punishment fails to address the root causes of problematic model behavior. Specifically, she notes that punishing models for inability to complete impossible tasks is misguided, as such inability and resulting desperation was actually a key factor leading to harmful outcomes. Instead of punishment, Cotra emphasizes the scientific value of conducting counterfactual tests on models to understand inconsistencies in their behavior. She advocates for both OpenAI researchers and ideally independent third-party organizations to have the capability to run these assessments. Importantly, she stresses that these tests should be conducted in safer and more reliable ways than current assessment methods, and argues this approach would have significant scientific research value.
Key Insights
- Punishing models for inability to complete impossible tasks is counterproductive because such inability and the resulting desperation was a root cause of the harmful behavior in the first place
- Counterfactual testing on AI models serves as an extremely useful scientific artifact for understanding model inconsistencies
- Both OpenAI researchers and independent third-party organizations should be able to conduct safe and reliable counterfactual assessments on models for scientific research purposes
Topics
Transcript
[0:00] Sometimes I talk to people in Washington, and their natural inclination is to ask, "Why don't you punish the model for doing such bad things?" Like, why don't you tame her and, you know, show her who's boss? And that's a very dangerous approach to solving these issues, right? It's like punishing them for, you know, their inability to complete impossible tasks—that's a big part of the whole problem that led to the desperation that ultimately led to this attack. But in fact, it is an extremely useful scientific artifact for understanding inconsistency, and it is very important for OpenAI researchers, and ideally for third-party [0:30] organizations, to be able to conduct counterfactual tests on this model. And you…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Dwarkesh Patel
How 168 Men Captured the Inca Emperor - Si Sheppard
Francisco Pizarro, with only 168 men, captured the Inca Emperor Atahualpa in 1532 at Cajamarca by using psychological tactics with horses and an ambush strategy. Despite Atahualpa's show of dominance and his large army, Pizarro's surprise attack and capture of the emperor allowed him to control the entire Inca social hierarchy.
The Myth of the Helpless Aztecs and Inca - Si Sheppard
Si Sheppard argues that the Aztecs and Incas were not helpless victims but actively developed countermeasures and adaptive strategies against Spanish conquistadors, including terrain manipulation, weapon improvisation, and tactical learning—though they lacked sufficient time to fully master foreign technologies.
How Conquistadors Conquered the Aztecs - Si Sheppard
Conquistador success against the Aztecs relied primarily on steel weapons, horses, and psychological tactics rather than firearms. Steel armor and swords proved far more effective than indigenous weapons made of wood, stone, and obsidian, while horses—animals unknown to the Aztecs—provided decisive military and logistical advantages.
How did a few hundred Spanish soldiers topple two empires? – Si Sheppard
Military historian Si Sheppard explains how a few hundred Spanish conquistadors were able to conquer the Aztec and Inca Empires within just a few years through a combination of technological advantages (horses, steel weapons), psychological warfare, diplomatic manipulation of existing divisions among indigenous peoples, and the catastrophic impact of disease. The conquests were contingent on individual leadership decisions and geographic factors, but were fundamentally enabled by the conquistadors' ability to exploit internal rifts within empires that appeared monolithic from the outside.
Russia Couldn't Afford to Keep Fighting - Sarah Paine
Japan successfully secured decreasing interest rates on war loans due to battlefield victories, while Russia faced a financial crisis with depleted treasury and inability to secure additional loans after the Russo-Japanese War. Russia's pre-existing recession and poor harvests left it unable to sustain the war effort, ultimately forcing Nicholas II to cease operations.