AI Tools Got Faster But Developers Didn't #ai #productivity #shorts
A METR study found that developers using AI coding tools were actually 19% slower at completing tasks, despite AI's faster generation speeds. The speaker explains this is due to workflow disruption and the need for human review of AI-generated code that looks correct but often isn't.
Summary
The speaker discusses a significant gap between AI capabilities and real-world implementation, particularly affecting both software developers and vendors. They reference a METR randomized controlled trial that found open source developers using AI coding tools completed tasks 19% slower, even when controlling for task difficulty, developer experience, and tool familiarity. The slowdown occurs because workflow disruption outweighs the benefits of faster code generation - developers spend significant time evaluating AI suggestions, correcting nearly-correct code, context switching between their mental models and AI output, and debugging subtle errors in generated code that appears correct. The speaker notes that 46% of developers in broader surveys don't fully trust AI-generated code, emphasizing these are experienced engineers, not technology resisters. They explain this phenomenon as a 'J-curve' that adoption researchers identify, where productivity initially dips when AI coding assistants are added to existing workflows before eventually improving. This productivity decline can last for months and occurs because tools change workflows without the workflows being redesigned around the new tools, creating a mismatch like 'running a new engine on old transmission.'
Key Insights
- A METR randomized controlled trial found that developers using AI coding tools completed tasks 19% slower, even when controlling for experience and task difficulty
- The speaker argues that workflow disruption outweighs AI's generation speed benefits because developers spend time evaluating suggestions, correcting code, and debugging subtle AI-introduced errors
- 46% of developers in surveys report not fully trusting AI-generated code, according to the speaker's cited research
- The speaker identifies a 'J-curve' adoption pattern where productivity dips before improving when AI tools are added to existing workflows
- The speaker claims there's an unprecedented gap between frontier AI capabilities and what actually happens in practice with vendors and developers
Topics
Full transcript available for MurmurCast members
Sign Up to Access