What If We Stopped Using GPUs? | YC Paper Club

The event begins with François, the moderator, introducing the core problem: GPU efficiency improvements have stalled over the past two years despite exponential growth in computational demands from transformer models. He explains that since 2020, NVIDIA shifted focus from energy efficiency to memory bandwidth and capacity due to attention's quadratic memory requirements. François proposes exploring alternative computing paradigms, particularly those inspired by the brain's remarkable 20-watt power consumption versus modern AI systems' massive energy demands.

François presents his own research on SPSA (Simultaneous Perturbation Stochastic Approximation), a gradient-free optimization method that uses finite differences rather than backpropagation. He argues this approach scales better with model parallelization and proposes 'Soma' (Shadow Mixture Optimization Collections), clustering datasets and distributing small expert models across GPUs worldwide. The key insight is that gradient estimation noise becomes worse with model size, but decreases when models are split—suggesting massive parallelization could work better without backpropagation.

Ilker presents optical computing, arguing that photons have 10,000 times lower losses than electrons over distance and massively parallel properties since photons don't interact. However, he highlights critical challenges: converting digital data to analog/optical and back wastes energy, passive optics struggle with nonlinear activation functions, and programming/calibration adds overhead. His team's work on diffusion models shows optical systems can achieve comparable results to digital networks with better energy efficiency for specific tasks, and they demonstrate power-law scaling benefits when adding optical parameters.

Alok discusses neuromorphic computing—using brain-inspired principles in chip design. He emphasizes the brain's remarkable co-adaptation of hardware and software over evolution, and notes that modern neural networks have strayed far from biological principles by adopting backpropagation and transformers. He categorizes neuromorphic research into three areas: energy efficiency through analog learning, event-driven computation using spike-based networks, and exotic substrates (photonics, memristors, coupled oscillators). Alok notes the field remains in research and development, with DMatrix showing promise by combining SRAM with logic in digital format.

Sean presents work on training biological neural cultures from rat brains to play Doom, treating the system as a black-box input-output dynamic system. Rather than manually encoding/decoding, they use PPO (Proximal Policy Optimization) with learned encoders that translate game states into electrode stimulation frequencies/amplitudes, and decoders that read spikes as actions. Feedback is provided through asynchronous (negative) and synchronous (positive) stimulation patterns, modulated by TD error. The work demonstrates that biological systems can learn complex tasks when given proper learning algorithms, though scalability to human-level intelligence remains unresolved.