Inference Chips for Agent Workflows

The transcript opens by challenging the assumption that inference hardware is a solved problem, arguing that existing GPU designs were built for simple prompt-in, response-out workloads rather than the complex, iterative loops that agentic AI systems require. Agents loop repeatedly, call external tools, branch and backtrack, and maintain context across dozens of steps — a fundamentally different computational pattern than traditional inference.

The speaker quantifies the inefficiency: current GPUs achieve only 30-40% of peak utilization on agentic workloads because the work is inherently bursty, alternating between memory-bound model calls, IO-bound tool use, and CPU-bound orchestration. This utilization gap represents the core business and technical opportunity for purpose-built silicon.

The transcript references major industry moves as evidence that the market recognizes this shift. Nvidia's $20 billion acquisition of Groq is cited as a signal that even the dominant GPU player sees agentic inference as a distinct hardware problem. Google's TPU v7, designed specifically for inference, is noted, though the speaker argues that no one has yet designed hardware specifically for the agent execution loop itself — features like fast context switching between models, native speculative decoding, and persistent KB-level caches across full execution graphs.

A key philosophical point is made about Groq: the speaker argues Groq's real innovation was not the chip itself but the compiler that made the chip usable. This insight is projected forward as a prediction — that the winning solution in agentic inference silicon will similarly depend on deep compiler and software-stack expertise, not hardware alone. The transcript closes as what appears to be a recruiting or investor pitch, inviting people who combine chip architecture knowledge with an understanding of agent execution to reach out.