How Open-Source AI Became Critical Infrastructure
Simon Mo, CEO of Infraact and lead maintainer of VLLM, discusses how open-source inference infrastructure has become critical to AI deployment, enabling enterprises to run frontier-quality open-weight models with greater control and cost flexibility than proprietary APIs. The conversation explores the economics of open-source models, licensing challenges, moderation trade-offs, and why open-source development remains essential despite the increasing compute requirements for training frontier models.
Summary
The episode examines the evolution of open-source AI from a niche enthusiast space to critical infrastructure. VLLM emerged in 2022 to solve fundamental challenges in serving large language models efficiently on GPUs—problems that differ significantly from traditional ML workloads due to variable input/output distributions, batching complexity, and the need for low-latency responses. The conversation traces when open-source became essential: early models like BERT required GPUs, but the true inflection point came around 2023 when applications like GitHub Copilot and ChatGPT made AI critical to daily work, prompting enterprises to seek control and flexibility through open-weight models.
The discussion highlights VLLM's role as critical infrastructure—an inference engine that optimizes thousands of model architectures across hardware vendors (NVIDIA, AMD, Google, etc.) and runs on roughly half a million GPUs globally. VLLM enables what proprietary vendors cannot: customizable performance tiers, with providers offering up to 10 different speed levels compared to the two modes (regular/fast) available with proprietary models. This control extends beyond speed to guardrails, data retention, security, and fine-tuning capabilities.
A significant portion addresses the economic sustainability of open-source models. Unlike open-source software, which relies on volunteer labor and company donations, frontier model training requires billions of dollars in compute. Recent models like Kimi K3 and Mistral have introduced licensing terms requiring commercial agreements above certain usage thresholds. Participants argue this reflects necessary economics rather than greed—without revenue mechanisms, governments rather than private labs would fund research, which would be worse for the ecosystem. The analogy to pharmaceuticals illustrates how revenue from deployed models should fund future R&D.
The conversation addresses moderation and control. When Hugging Face used a Chinese open-source model to contain a rogue unsandboxed OpenAI model during testing, it illustrated how proprietary guardrails create false positives that block legitimate use cases. The speakers argue that if moderation cannot be perfectly solved (a historically intractable problem), people will default to open-weight models where they can control guardrails. Infraact developers themselves switched from Claude to Kimi K3 because Claude's content filters block legitimate GPU kernel research with false positives that cause work loss.
On capability gaps, Simon argues there is no meaningful difference between frontier closed and open models today—the distinction is distribution and go-to-market strategy rather than inherent capability. What differentiates open-weight labs is not data source but the environments built around models for iterative improvement. Moonshot's success with Kimi K3 on coding tasks came from constructing superior front-end rendering environments for recursive self-improvement, not from distilling proprietary models. Participants suggest distillation is not a critical component of current Chinese model progress; instead, innovation stems from smart researchers making algorithmic choices like removing rotary positional embeddings.
The transcript concludes with broader implications: the centralization of AI inference under proprietary APIs mirrors social media's consolidation, but without legal exemptions from content liability. Open-source models provide a check on this consolidation and enable global collaboration where researchers worldwide learn from and improve upon each other's work.
About this episode
Elena Burger and Matt Bornstein are joined by Simon Mo, co-founder and CEO of Inferact, the open-source inference engine powering many of today's most advanced AI applications. Together, they explore how open-source AI evolved from a research project into critical infrastructure, why inference has become one of the most important layers of the AI stack, and what it takes to bring frontier intelligence to developers around the world. The conversation covers vLLM's origins, the rise of open-weight models, why companies increasingly want control over their AI infrastructure, and how open-source inference enables the next generation of AI applications. They also discuss model licensing, the economics of open-weight AI, Kimi K3, distillation, AI infrastructure, and why Simon believes the gap between open and closed models is rapidly disappearing.
Key Insights
- Simon Mo argues that serving large language models differs fundamentally from traditional ML workloads because of variable input/output distributions, non-deterministic outputs, and the complexity of batching and scheduling at the inference layer, requiring specialized engineering that did not previously exist.
- The speakers claim that open-source AI became critical infrastructure around 2023 when applications like GitHub Copilot made AI essential to daily work, prompting enterprises to seek control and flexibility that proprietary APIs could not provide.
- Mo contends that VLLM enables customizable performance gradations unavailable with proprietary models—up to 10 different speed levels versus two modes with closed alternatives—allowing users to calibrate cost and performance precisely.
- The participants argue that open-source model licensing terms requiring commercial agreements above usage thresholds reflect economic necessity rather than greed, since training frontier models costs billions and requires revenue mechanisms to fund future research; without this, government funding would replace private labs.
- Simon Mo asserts there is no meaningful capability gap between frontier closed and open models today; the difference is in distribution strategy and go-to-market approach rather than underlying capability, contradicting the perception that closed models inherently lead.
- The speakers propose that moderation is historically unsolved and likely unsolvable, making false positives inevitable in proprietary systems, which will drive users toward open-weight models where they can control guardrails for their specific use cases.
- Mo claims that what differentiates open-weight model labs today is not data access but the quality of environments constructed for iterative improvement—exemplified by Moonshot's front-end rendering capabilities for coding tasks—which cannot be replicated through distillation.
- The participants argue that distillation is not a critical component of current frontier model progress; instead, innovation stems from research teams making novel algorithmic choices, such as removing previously necessary components like rotary positional embeddings, based on deeper understanding.
Topics
Transcript
The fun thought experiment is if GPUs dropped in price by 99%, then do we get back to a real open source world? If moderation is never solved, in the future people will go to OpenWay by default. Because that is where you know for sure you can control your guardrail for trusted use cases. And can you talk about where VLLM sits in that stack? VLLM is an inference engine. It is kind of like databases and operating system and other critical software to power AGI, NVIDIA, AMD, Google. Their newest chip will make sure VLM can run on them. And then a lot of cases they use VLM as a benchmark while bridging almost a 10x gap. For…
Full transcript available for MurmurCast members
Sign Up to AccessMore from The a16z Show
The Top 100 Consumer AI Apps: Who’s Actually Paying?
A16Z's seventh edition Top 100 Consumer AI Apps Report reveals that while half of Americans use AI, only 4.5% pay for subscriptions, with spending heavily concentrated among developers and power users. The report highlights emerging personal agents as a major trend, significant divergence between ChatGPT, Claude, and Gemini, and substantial untapped opportunities in categories like dating, social, shopping, and entertainment.
David George & Jack Altman on AI, Autonomy, and the Next $25 Trillion
David George from Andreessen Horowitz argues that AI, autonomy, and robotics will create over $25 trillion in market value over the next decade, with success across multiple layers of the stack—frontier models, open source, and applications will all flourish due to massive supply constraints and early-stage demand penetration among knowledge workers.
Beyond the God Model | Alex Atallah & Amjad Masad
Alex Atallah (OpenRouter) and Amjad Masad (Replit) discuss how the future of AI development lies in specialized, diverse models working together rather than relying on single AGI-like models. They argue that enterprises need independence from foundation model providers through model diversity, cost efficiency, and ownership of their own AI capabilities, while also addressing the risks of general-purpose agents and the importance of model specialization.
Why AI Agents Can Beat the Incumbents
Leo, an AI-native startup, is building multi-agent systems to automate end-to-end procurement processes for enterprises by coordinating work across multiple departments, stakeholders, and external systems that traditional procurement software cannot handle. The company differentiates itself from incumbents by owning the entire job workflow rather than being confined to a single system of record, and builds customer trust through human-in-the-loop approaches before scaling to fully autonomous negotiations.
The $1 Trillion AI Buildout | State of Markets
A16Z partners discuss the $1 trillion AI infrastructure buildout, arguing it's driven by real earnings growth rather than inflated valuations, with adoption still extremely early at the enterprise level. They highlight opportunities across consumer agents, robotics, autonomous vehicles, and enterprise diffusion, while noting that the market's 90% gain since ChatGPT reflects fundamental business performance rather than speculative excess.