New LLMs Are Unlocking Robot-Use Agents

The conversation begins by establishing the surprising generalization capabilities of coding agents across domains, leading to their application in robotics. Jamie and Vincent from Waddle Labs explain their approach to creating LLMs that drive robots through efficient system design and data collection, while Jay from RoboCurve describes their role as an evaluation company for physical AI across various robot morphologies and control paradigms.

The discussion traces the evolution from RT2—which fine-tuned language models to output end-effector poses instead of English—to modern VLAs (Visual Language-Action models) that benefit from the architectural capabilities of large language models. A key insight emerges around the shift from fine-tuning to in-context learning: rather than baking knowledge into model weights, modern LLMs can write complex policies as code with sufficient reasoning and tool use capabilities.

François introduces a framework distinguishing between different learning paradigms: learning embedded in weights (through training), in-context learning (ICL), and various middle grounds like LoRA fine-tuning. Experiments show ICL improves cheaply but saturates around 20-40 examples and breaks down beyond the model's context window. This motivates the strategy of consolidating context-learned skills into tools or policies that can be reused and refined—akin to how agents write and refactor code libraries.

The concept of 'code as policy' is central to the discussion, building on work like Voyager where agents create tools on-the-fly using Python functions. Early papers from Google DeepMind (Coda) demonstrated that LLM agents could write complex robot control code given lists of primitive functions, without additional robot-specific training data. Modern models like Claude/Astra dramatically improve on this by leveraging vast pre-training on web data, images, and especially computer usage data (CAD, Blender interactions, simulations).

The speakers connect this progress to the Platonic Representation Hypothesis—the idea that models trained on diverse data converge toward consistent world representations. This explains why language models can suddenly become effective robot controllers: a sufficiently capable language model implicitly learns the spatial and physical understanding needed for robotics. Astra's improvements are attributed to pre-training on massive computer usage datasets, which teach spatial reasoning applicable to physical control.

Looking forward, the team predicts general-purpose robots within 2 years capable of performing tasks any competent teenager could do with bare hands. Key challenges include reducing latency (currently bottlenecked by model inference speed) and managing the consolidation phase—similar to how biological systems use sleep to compress memories into skills. They envision a cycle where models learn in context, compress skills during 'offline' phases, and refactor growing skill libraries for faster deployment, drawing parallels to Dream Coder and classical data aggregation approaches like Dagger.