Why Inference Matters 12x More Than Training | YC Paper Club

François Maillet opens the discussion by establishing the economic case for inference optimization: DeepSeek reports 6x more FLOPs spent on inference annually per model weight, scaling to 12x over a 2-year model lifetime. This ratio suggests ML papers should prioritize minimizing inference FLOPs over training FLOPs, yet most benchmarking remains fixated on training FLOPs. DeepSeek's reported 80% margins demonstrate the competitive advantage of inference innovation.

Dan Fu from Together AI walks through the complete inference request lifecycle: routing requests to appropriate GPU replicas considering KV cache locality, tokenization with tool-calling complexity, prefill (compute-bound prompt processing) versus decode (memory-bound token generation), and the KV cache retention strategies that enable high cache hit rates. He emphasizes three optimization levers: (1) kernel optimization using agents to automatically optimize inner loops, (2) parallelism schemes leveraging larger-scale interconnects like GB racks to serve 72 GPUs cohesively with expert parallelism, and (3) quantization with rigorous evaluation loops like Flappy Bird testing to ensure output quality. Together successfully helped MiniMax quantize their model to NVFP4 while maintaining performance.

Jeff Morgan from Ollama addresses the quality consistency problem across inference providers. Research shows the performance gap between the worst and best provider serving the same model exceeds the gap between open and closed models—a critical discovery. Different providers vary in model precision, tool calling accuracy, sampling techniques, and inference engine implementations. Jeff outlines monitoring approaches: running evals periodically, using distribution tests comparing outputs to reference weights (~$1 per test), log probability monitoring to detect serving stack changes, and output shape introspection to catch issues like infinite loops. He argues that quantization post-launch differs from labs sharing their own quantized versions pre-launch, and proposes that monitoring LLM quality in production represents an underexplored startup opportunity.

Joe Fiati from Luminal explains their compiler approach using equality saturation (e-graphs) rather than traditional rewrite rules. They reduce all PyTorch operations to 14-15 primitive ops (add, multiply, sum) creating a "PyTorch complete" bottleneck that avoids the combinatorial explosion of mapping 2,000 PyTorch ops to diverse hardware. Their multi-level IR explicitly represents GPU hierarchy (thread, warp, thread block, GPU, node levels), enabling combination of vendor libraries (cuBLAS, CUTLASS, ThunderKit) into mega kernels or separate kernels. Rather than building complex cost models, they profile actual hardware execution—accepting that measuring everything on real hardware across pruned search spaces outperforms predictive models. This pragmatic approach commoditizes heterogeneous hardware (Nvidia, AMD, TPU, Trainium, inference ASICs).

Lang Shang from SG-Lang traces the evolution from Radix attention solving agent workload prefix reuse challenges to their unified inference engine. Original Radix attention managed KV cache as a tree structure for shared prefixes across parallel sub-agents. Mid-2024 v0.2 generalized this across model architectures. Key innovations include: overlap scheduling eliminating CPU-GPU synchronization bottlenecks (2x throughput gains on Blackwell), hierarchical memory (L1/L2/L3 cache across GPU/host/storage reducing time-to-first-token by 56%), and unified radix trees coordinating heterogeneous memory pools for full attention tokens, sliding window tokens, and recurrent states. The unified memory pool flexibly allocates physical memory across different component needs without pre-set ratios, using compaction operations to reorganize fragments. This unified approach addresses the complexity of supporting diverse context lengths and modern architectures (sliding windows, SSMs, recurrent states) simultaneously.