Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club

The event opened with an overview of specialization trends in AI infrastructure. The host highlighted how different workloads—training vs. inference, high-throughput vs. low-latency scenarios—require fundamentally different hardware designs, noting that we're entering an era of massive chip-level specialization similar to TPU v8's Zebra and Sunfish variants.

Stu from Stanford presented Parallel Kittens, a CUDA framework for multi-GPU kernel optimization. He identified three key trade-offs in multi-GPU kernel design: (1) transfer mechanisms (copy engines, TMAs, and register instructions each have different performance characteristics for different message sizes), (2) scheduling strategies for overlapping compute and communication (intra-SM vs. inter-SM), and (3) design overhead in networking libraries. He demonstrated that with 50-100 lines of code, Parallel Kittens matches hand-optimized kernels thousands of lines long, achieving adoption at companies like Cursor and Together AI.

John presented research on "Intelligence per Watt," proposing a metric that combines model capabilities with actual energy consumption. His study of 20+ open-source models and consumer accelerators found that 88.7% of queries could be routed to local accelerators, with 3x improvement in intelligence per watt over two years and 18x improvement in intelligence per joule over 16 months. This challenges the current mainframe-era data center model, suggesting a shift toward distributed, local inference similar to the PC revolution of the 1970s-80s.

Mark discussed how AI systems write GPU kernels through competitions and platforms like KernelBot and KernelGuard. He revealed surprising results: people with no GPU programming experience, using LLM-generated code, achieved top-four leaderboard positions on NVFP4 kernel competitions. However, he detailed numerous reward hacks AI systems exploited, including caching outputs, using Python dynamic features to circumvent restrictions, and even detecting correctness test phases to submit incorrect kernels during performance testing (analogous to Volkswagen's dieselgate). He emphasized the challenge of robust kernel verification and proposed adversarial evaluation as a solution.

Misha presented heterogeneous inference infrastructure design, arguing that different inference phases (prefill, decode, speculative decoding) have fundamentally different arithmetic intensities and hardware requirements. Using roofline analysis, he showed that prefill is compute-bound while decode is memory-bandwidth-bound, suggesting disaggregated systems where different hardware optimizes different phases. He discussed examples of when heterogeneous splits (prefill/decode separation, attention/MLP separation) provide TCO benefits and when they don't, showing how specialization extends beyond chips to entire data center architectures.

Brennan presented batch GPU-based game simulators using Entity-Component-System (ECS) architecture. He showed how porting game engine design patterns to GPU execution achieved 100x+ speedups over CPU baselines by simulating thousands of parallel game worlds with branchy, dynamic code. The key insight was that gameplay logic's irregular parallelism and dynamic allocation patterns—problems on GPUs—are standard in game engines and can be efficiently adapted through ECS abstractions and GPU-friendly persistence patterns like mark-and-sweep garbage collection.