Can Open Source Keep AI Power From Concentrating?
Lucas Kaiser, co-author of the Transformer paper, discusses how AI power is currently concentrating in large companies due to the resource-intensive nature of current technology, but argues this is not inevitable. He believes research breakthroughs in algorithms and training methods could enable smaller players and distributed models to compete effectively.
Summary
In this episode of the A16Z podcast, host Sophia Du interviews Lucas Kaiser about the concentration of AI power and whether open source can create a more distributed future. Kaiser explains that current AI concentration stems from practical technological requirements rather than fundamental inevitability. Today's dominant architecture, the Transformer, requires enormous amounts of data and compute resources, which only large companies can afford. This has led to a business model where companies train expensive models on massive data centers and charge users for access. However, Kaiser emphasizes that Transformers are less than a decade old and represent just one approach to AI. He argues that humans demonstrate the existence of an alternative model: expertise distributed across specialized domains rather than concentrated in single all-knowing systems. Kaiser points out that current large language models struggle with specialized tasks and domain expertise compared to human experts, suggesting room for algorithmic innovation. He notes that while transformers excel when trained on internet-scale data, they perform poorly on smaller, specialized datasets—a limitation that may be solvable through research breakthroughs in loss functions, training methods, and data approaches rather than just scaling up. Kaiser expresses optimism that increased cost of scaling will redirect research focus toward fundamental breakthroughs. He also highlights democratization of compute: a single RTX 5090 GPU now contains more power than the eight GPUs used to develop Transformers, enabling researchers outside major labs to experiment and innovate. He concludes that the current concentrated state is a temporary phase driven by convenience and technological maturity, not an inherent feature of AI.
About this episode
MTS host Sophia Dew visits the Open Source AI Summit in San Francisco to ask researchers and founders across the AI stack a central question: can open source prevent AI power from concentrating in the hands of a few companies? Lukasz Kaiser, co-author of Attention Is All You Need, argues that today’s concentration may be a feature of the current technological paradigm rather than a permanent feature of AI. Transformers reward enormous amounts of data and compute, but future breakthroughs could make smaller, more specialized models far more capable. Across conversations with researchers and builders working on open models, infrastructure, and applications, Sophia explores why China has taken the lead in open-weight models, whether the U.S. needs more open-model startups, what it means for companies to own their own intelligence, and where openness alone falls short, particularly when access to compute remains concentrated.
Key Insights
- Kaiser argues that current AI concentration is a property of transformer technology and current business practices, not an inevitable feature of AI itself, as evidenced by humans being proof that distributed expertise can exceed generalist performance.
- He claims transformers require internet-scale data to function effectively but perform poorly on specialized datasets, suggesting research breakthroughs in training methods and loss functions—not just scaling—could enable smaller players to build effective specialized models.
- Kaiser observes that while large companies prioritize product development and scaling, increasing costs of training giant models may redirect research focus back toward fundamental algorithmic innovations, creating opportunities for open source projects and academic researchers.
Topics
Transcript
AI is becoming more powerful, but the resources needed to build it are increasingly concentrated. Does it have to stay that way? MTS host Sophia Du heads to the Open Source AI Summit in San Francisco to ask researchers and founders across the AI stack whether open source can create a more distributed future for AI. Lucas Kaiser, co-author of the landmark Attention is All You Need paper, argues that today's concentration may be a property of our current technology, not an inevitable feature of AI. Transformers thrive on enormous amounts of data and compute, but they're less than a decade old, and the next breakthrough could change those economics. From open models and access to compute, to companies owning…
Full transcript available for MurmurCast members
Sign Up to AccessMore from The a16z Show
Your AI Doctor Is Coming | Julie Yoo
Julie Yoo, a healthcare investor at Andreessen Horowitz, argues that AI will benefit healthcare more than any other industry because healthcare has historically underinvested in technology, allowing it to leapfrog legacy systems and adopt AI-native solutions directly. She identifies major opportunities in consumer healthcare, AI-native care delivery, robotics, and new payment models, predicting a future where individuals have personalized AI doctors available continuously.
Aaron Levie on Why Open AI Wins
Aaron Levie, CEO of Box, discusses why open-weight AI models benefit the entire AI ecosystem rather than threatening frontier labs, argues that the economic value in AI accrues to inference infrastructure rather than model weights, and explains why model routing will become the default enterprise AI strategy.
Fei Fei Li: The Race to Build World Models For AI
World Labs launched Atlas, a new world model built on next-view prediction that unifies 3D reconstruction and generation. The model can create spatially-grounded video frames from sparse input images (as few as three), reducing the data requirements for 3D scene capture by 50-100x, with applications ranging from creative content to robotics simulation.
The $100B Niches Hiding Inside Payments
Max Levchin and Alex Rampell discuss 25+ years of payments innovation, tracing PayPal's origins through their journey building Affirm. They explore how the credit card remains the best payment interface ever created, why there are no niches smaller than $100B in payments, and how AI may finally be ready to reinvent the payment experience itself.
Inside Moderna’s Personalized Cancer Vaccine
Moderna and Merck announced positive Phase 3 results for an individualized mRNA cancer vaccine (Intesmiran) for melanoma, achieving 80% disease-free survival at five years compared to 60% with Keytruda alone. The breakthrough combines mRNA technology with personalization, sequencing each patient's tumor to identify mutations and create a patient-specific vaccine that teaches the immune system to recognize cancer cells.