Chelsea Finn: This is the State of the Art in Robotics

Chelsea Finn, founder of Physical Intelligence, presents the state of physical intelligence and robotics, focusing on how to develop general-purpose robots capable of performing diverse tasks reliably in the real world. She begins by contextualizing the challenge: unlike language models where users can tolerate mistakes, physical robots operating autonomously must achieve high reliability (>90%) because they directly affect the physical environment.

Finn introduces a scalable reinforcement learning approach that improves upon standard algorithms like PPO by addressing key inefficiencies. The method includes human interventions to prevent dead-end trajectories and a general-purpose value function trained on diverse robot experience data to estimate success, reducing the number of attempts needed for learning. She demonstrates this through examples like making lattes and building cardboard boxes, achieving substantial throughput improvements through reinforcement learning post-training.

A critical innovation Finn highlights is implementing memory at multiple timescales for long-horizon tasks. Short-term video memory (10 seconds) is computed efficiently, while longer-term memory (10-15 minutes) is represented as compressed text summaries. This enables robots to autonomously complete complex multi-step tasks like kitchen cleaning without repetition.

Finn then discusses the development of a general-purpose foundation model (PIO7) trained on diverse, heterogeneous data including low-quality demonstrations, policy rollouts, human videos, and web data. The key technical unlock was detailed prompting that provides the model with context including instructions, subtask descriptions, metadata about data quality, and subgoal images. This enables the model to leverage even low-quality data effectively.

The PIO7 model matches or exceeds the performance of task-specific fine-tuned models across multiple domains without post-training, representing a shift from the pre-training/fine-tuning paradigm toward out-of-the-box general models. Finn demonstrates compositional generalization—the model can interact with rarely-seen objects (air fryers) and transfer skills across different robot platforms (cloth folding on an unseen BiARM UR5E robot) despite lacking specific training data.

Finn positions robotics on a 15-year timeline of AI development, arguing that robotics has progressed from collecting bespoke datasets (2023, analogous to 2012 ImageNet) to the GPT/DALI era (2024-2025) in just a few years. She notes that models are now deployed in real-world companies (Ultra, Weave) for practical applications, not just research demonstrations.

In the Q&A, Finn addresses when teams should adopt generalist policies, noting that starting with fine-tuned open-source models like PIO5 is recommended immediately. She discusses control output formats (joint positions versus 3D end-effector positions), the role of imagination/future prediction in the models, and speed improvements needed for practical deployment. She emphasizes that robots require autonomous experience on physical platforms and cannot rely solely on human video data, highlighting an important limitation compared to language models trained on internet text.