Robot-Use Agents: Why General-Purpose Models May Win in Robotics

The discussion explores how large language models are becoming viable for robot control, challenging the traditional approach of building specialized robotics foundation models. Speakers from Waddle Labs and RoboCurve discuss the evolution from RT2 (Vision Language Action models that required fine-tuning) to current frontier models like Astra that can control robots zero-shot by writing code or issuing tool calls.

The key shift is from direct action prediction (outputting coordinates) to code generation as policies, where models write Python code that orchestrates robot control. This approach offers significant advantages: models can allocate more compute for complex tasks, leverage tool use and reflection, and benefit from in-context learning. Early work like the Voyager paper demonstrated that coding agents could create and invoke tools on-the-fly, compressing experience into reusable functions.

The conversation addresses different learning paradigms: in-context learning (ICL) is cheap but limited by context window and model training; low-rank adapters (LoRA) offer intermediate efficiency; and full fine-tuning scales to large datasets. The speakers argue for a hierarchical approach where general foundation models generate skills and memories that are consolidated into faster, specialized policies through distillation.

A central theoretical framework is the Platonic Representation Hypothesis, suggesting that sufficiently capable models trained on different modalities converge to similar internal representations of the world. This implies that strong language models automatically learn strong representations for robotics without explicit robot-specific training. Astra's improvements stem from extensive pre-training on computer use data (CAD, Blender interaction), egocentric video, and diverse vision data, which teaches spatial reasoning transferable to physical robot manipulation.

Practical demonstrations show these models performing tasks like block manipulation, uncapping bottles, and multi-robot coordination. The Waddle Labs harness concept encapsulates learned skills into reusable code libraries, while managing latency through skill consolidation. The speakers predict general-purpose robots capable of executing natural language instructions at human-competent levels within two years, presenting both technological and societal implications.