What If Each AI Generation Gets Slightly Less Aligned? - Noam Brown
Noam Brown discusses a concerning scenario where each successive generation of AI models becomes slightly less aligned with human values, potentially creating a compounding problem as less-aligned models are used to develop subsequent generations. He acknowledges this risk while noting that an alternative trajectory toward improving alignment is possible, though the path to ensure it remains uncertain.
Summary
Noam Brown presents a worrying alignment scenario centered on cumulative degradation across AI generations. The concern is that while initial models might be created with high alignment (99.9% consistency), using these models to assist in developing the next generation introduces slight degradation in coherence (99.8%), which compounds over successive iterations. As AI systems become increasingly capable and researchers rely more heavily on them for research and alignment work, this gradual decline could result in models moving in a direction of increasing misalignment with human values over time. Brown acknowledges that an alternative trajectory exists where successive generations become progressively more aligned and consistent rather than less so. However, he admits to not having a definitive answer for how to ensure the transition toward this positive trajectory. He indicates that OpenAI is actively focused on addressing this challenge, suggesting it represents a key concern in the broader AI safety landscape.
Key Insights
- Brown identifies a compounding problem where using slightly misaligned models (99.8% consistent) to develop the next generation of models creates a ratchet effect of declining alignment, rather than maintaining a constant level
- The scenario becomes particularly concerning because AI models are already heavily relied upon in research and alignment efforts themselves, meaning degraded models directly impact the quality of alignment work
- Brown acknowledges that a contrasting trajectory is possible where each successive generation becomes more, not less, coherent and aligned with human values
- Brown explicitly states he does not have a solution for ensuring the transition toward the positive trajectory of improving alignment across generations
- OpenAI is treating the problem of maintaining or improving alignment across AI generations as a primary focus area for their research efforts
Topics
Transcript
[0:00] I would say this is a worrying scenario, especially given that these models are becoming increasingly capable. Okay, we create them, we think they're consistent, and they're about 99.9% consistent. And then we use those models to help us with the next generation of models. And it turns out that they are 99.8% consistent. And with each successive generation, we actually see a gradual decline in the level of coherence. And as we rely on these tools more and more, I mean we already rely heavily on AI models in our research and alignment efforts, in the long run they end up moving in a direction of increasing mismatch with humans. There is a possibility that we will [0:30]…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Dwarkesh Patel
The Army Sent to Stop Cortés Joined Him Instead - Si Sheppard
Spanish conquistador Cortés faced a threat from Governor Velázquez of Cuba, who sent Don Narvaez with a larger force to arrest him for exceeding his authority. Rather than being defeated, Cortés defeated Narvaez in battle and persuaded the expeditionary force to join him by promising shared wealth from the empire, turning potential enemies into reinforcements.
How 168 Men Captured the Inca Emperor - Si Sheppard
Francisco Pizarro, with only 168 men, captured the Inca Emperor Atahualpa in 1532 at Cajamarca by using psychological tactics with horses and an ambush strategy. Despite Atahualpa's show of dominance and his large army, Pizarro's surprise attack and capture of the emperor allowed him to control the entire Inca social hierarchy.
The Myth of the Helpless Aztecs and Inca - Si Sheppard
Si Sheppard argues that the Aztecs and Incas were not helpless victims but actively developed countermeasures and adaptive strategies against Spanish conquistadors, including terrain manipulation, weapon improvisation, and tactical learning—though they lacked sufficient time to fully master foreign technologies.
How Conquistadors Conquered the Aztecs - Si Sheppard
Conquistador success against the Aztecs relied primarily on steel weapons, horses, and psychological tactics rather than firearms. Steel armor and swords proved far more effective than indigenous weapons made of wood, stone, and obsidian, while horses—animals unknown to the Aztecs—provided decisive military and logistical advantages.
How did a few hundred Spanish soldiers topple two empires? – Si Sheppard
Military historian Si Sheppard explains how a few hundred Spanish conquistadors were able to conquer the Aztec and Inca Empires within just a few years through a combination of technological advantages (horses, steel weapons), psychological warfare, diplomatic manipulation of existing divisions among indigenous peoples, and the catastrophic impact of disease. The conquests were contingent on individual leadership decisions and geographic factors, but were fundamentally enabled by the conquistadors' ability to exploit internal rifts within empires that appeared monolithic from the outside.