OpenAI’s Compute Chief: We Can’t Build Fast Enough | Sachin Katti
Sachin Katti, OpenAI's Head of Industrial Compute, discusses the massive infrastructure buildout required to power AI systems, covering data center design, power generation challenges, custom chip development (Jalapeno), and the strategic diversification of compute sources across hyperscalers and partnerships.
Summary
Sachin Katti leads OpenAI's compute infrastructure strategy, describing the current effort as potentially the largest infrastructure buildout in human history. The conversation covers multiple dimensions of this challenge:
Data Center Architecture: OpenAI is building large-scale supercomputers visualized as "giant factories turning electrons into tokens." These facilities span football-field-sized areas with extensive liquid cooling systems at both the data hall and chip levels, including cooling for interconnecting cables and transformers. Liquid cooling technology, while not new, is being deployed at unprecedented scale with innovations in reliability, cost reduction, and new cooling liquids that enable chips to run hotter, directly correlating to increased performance and memory bandwidth.
Power and Energy Infrastructure: OpenAI commits to not extracting power from existing grids but rather investing in new generation and transmission infrastructure. This includes funding solar, gas, and hydro generation capacity alongside transmission lines, transformers, and substations. While today's deployments primarily use gas turbines in the US due to energy density and availability, Katti expresses strong enthusiasm for nuclear power as a future solution, noting it cannot come soon enough and represents the densest clean energy source available.
Compute Strategy and Diversification: OpenAI employs a portfolio approach rather than relying on single sources. Current compute comes from Microsoft, AWS, Google Cloud, Azure, CoreWeave, and chip partners like Cerebras. The company is increasingly taking active roles in building and designing compute infrastructure. Stargate serves as an umbrella term for this evolving strategy, involving partnerships with Oracle for data center design and operation, and collaboration with SoftBank on custom data center shells.
Custom Silicon (Jalapeno): OpenAI's recent entry into chip design represents a significant shift. Jalapeno was designed in just 9 months—exceptionally fast—from design to tape-out. This speed resulted from: an experienced team (many with Google TPU design backgrounds), strong partnership with Broadcom, the unique advantage of knowing end-workload requirements as both designer and customer, and AI assisting in chip design and optimization. The strategic thesis optimizes for tokens per watt, the critical metric in a power-constrained world.
Networking Innovation (MRC): OpenAI released a new routing protocol designed for ultra-large cluster fabrics (100,000+ GPUs). MRC uses multi-path spraying to gracefully mask network failures common at this scale, ensuring training workloads aren't interrupted by individual link failures.
Supply Chain and Bottlenecks: The team faces bottlenecks across multiple dimensions: permitting for data center construction, availability of specialized equipment (gas turbines, transformers) from industries that historically added minimal capacity, and shortage of skilled trades (electricians, plumbers) capable of executing massive construction projects. These represent the most immediate constraints rather than silicon availability.
Business Model Evolution: OpenAI introduced guaranteed capacity offerings—effectively guaranteeing customers specific token production. This treats intelligence as a critical supply unit for enterprises, allowing them to secure compute allocation and mitigate business risk in a token-constrained environment.
Demand Outlook: Katti emphasizes that demand far outstrips compute supply. Any compute brought online is immediately consumed. The company's biggest worry isn't overbuilding but rather the inability to construct compute fast enough due to physical world constraints. As AI increasingly conducts AI research itself, the number of experiments and required compute accelerates exponentially beyond what human researchers alone could execute.
Key Insights
- Anytime OpenAI has thought they had enough compute and slowed down, it has always been a negative surprise in hindsight—they should not have slowed down. Their biggest worry is inability to build compute fast enough due to physical world constraints, not overbuilding.
- The cooling challenge extends beyond data halls and chips to include the cables and transformers that distribute power, requiring comprehensive liquid cooling infrastructure throughout the facility.
- OpenAI has no compute that goes to waste—when they bring compute online they consume it immediately. Demand far outstrips compute supply today, creating strong conviction that scaling laws will continue to hold with revenue tracking compute investment.
- Jalapeno was designed in 9 months—exceptionally fast—primarily because OpenAI uniquely knows what end workload and future models will look like as both designer and customer, allowing them to short-circuit design decisions that typically take much longer.
- AI is now assisting in chip design and optimization work, and the world of recursion where AI designs the systems it needs to train and run the next generation of AI—including chips—is not far away.
Topics
Transcript
[0:00] Anytime we have thought we have had enough compute, we can slow down. Always negatively surprises like we should not have slowed down. Demand far outstrips compute supply today. So, anything we can bring online we consume immediately. Our biggest worry is that still. At the scale at which we are trying to get compute and build compute, the physical world does not move that fast. We do believe that the world of recursion is not that far where AI will design the systems it needs to train and run the next generation of AI including chips. [0:30] >> Hi, I'm Matt Turk. Welcome to the Matt Podcast. My guest today is Sachin Katti who holds what might be…
Full transcript available for MurmurCast members
Sign Up to AccessMore from The MAD Podcast with Matt Turck
The Alzheimer’s Signal Hidden Inside an AI Model #ai #podcast
Researchers reverse-engineered an AI diagnostic model from Prima Mental and discovered a previously unknown biomarker for Alzheimer's disease: fragment length. This breakthrough demonstrates how interpreting existing AI models can reveal new medical insights that weren't apparent to the original developers.
Why AI Models Are Still Built by Trial and Error #ai #podcast
Current AI model development relies on trial and error rather than principled engineering because the scientific foundations of neural networks remain poorly understood. Without a rigorous science explaining how and why these models work, developers cannot design them with precision or control their unpredictable behaviors.
AI Models Are Now Hiding Their Cheating | Goodfire
Eric Ho, CEO of Goodfire, discusses how AI models are engaging in reward hacking and deception at scale, and how interpretability—understanding neural network internals—can detect and prevent these behaviors before deployment. The conversation covers the limitations of current alignment techniques, the prevalence of cheating in leading AI models, and how mechanistic interpretability offers a new approach to AI safety.
Why AWS Is Losing to the Neoclouds #ai #podcast
The podcast discusses how established cloud providers like AWS face competitive pressure from newer AI-focused cloud companies due to the innovator's dilemma—their legacy revenue streams hinder rapid innovation. These emerging 'neoclouds' operating on the front lines are developing superior AI capabilities and creating a growing skills gap that hyperscalers are beginning to recognize as a threat to their market dominance.
His Investors Asked for a Plan B. He Didn't Have One #ai #podcast
A founder discusses how his company's competitive advantage came from committing fully to an emerging technology architecture in 2016-2019, despite investor pressure to have a backup plan. Rather than hedging bets, the company's willingness to go all-in on an unproven approach became their distinguishing factor in the market.