Fei Fei Li: The Race to Build World Models For AI

Fei-Fei Li, Justin Johnson, and Ben Mildenhall from World Labs discussed their new frontier model Atlas, which represents a fundamental shift in how AI approaches spatial intelligence. Unlike previous models built on next-token prediction (LLMs) or next-frame prediction (video models), Atlas is built on next-view prediction—given some views of a scene, it predicts what the scene looks like from any arbitrary point in space and time.

Atlas combines two traditionally separate computer vision tasks: dense 3D reconstruction and generative pixel synthesis. Previously, reconstruction required exhaustive capture of a scene with hundreds of images to ensure every angle was covered, making it impractical for most applications. Atlas reduces this requirement dramatically—it can reconstruct complex scenes from as few as three images, representing a 50-100x reduction in capture requirements. This is achieved by combining triangulation from multiple views (reconstruction) with generative filling of occluded regions (generation).

The model is multimodal by design, natively working with text, images, videos, camera poses, and depth maps as inputs. Camera poses are a critical native input—every frame has an associated 3D camera position and parameters, allowing spatial grounding that distinguishes Atlas from other video generation models that lack this geometric precision.

Key capabilities include camera-conditioned generation (steering viewpoints through space), sparse 3D reconstruction (turning multi-view captures into navigable 3D scenes), and simulation including the famous bullet-time effects from The Matrix. Traditionally requiring hundreds of cameras and green screen studios, these effects now work with just three iPhones on tripods.

The team explained that achieving this required moving beyond their previous product Marble, which output Gaussian splats and couldn't gracefully scale to handle many input images or extensive context windows. Atlas bifurcates modalities earlier in the architecture and treats new-view prediction as the fundamental primitive, allowing it to output either RGB frames, 3D reconstructions, or Gaussian splats depending on the use case.

On scaling, the team reported being at the beginning of the scaling curve for this architectural approach, limited primarily by compute rather than data or architecture. Each increase in model size and training time showed significant improvements. Ben Mildenhall's conviction in scaling came from witnessing dense reconstruction improvements over three years prior to the company's founding.

For creative applications, Atlas extends workflows where artists compose multi-stage pipelines. Rather than requesting different viewpoints from multiple models and dealing with inconsistency, they can now ground generations in a persistent 3D world with precise viewpoint control. Use cases extend to architecture, design, and any field requiring virtual replication of physical spaces.

The robotics application is particularly significant. World Labs acquired Cynex to address the critical bottleneck in robotics: data collection for sim-to-real transfer. Dense reconstruction of robot environments is currently laborious and slow. Atlas dramatically accelerates the real-to-sim step, enabling faster training data generation. The vision extends beyond reconstruction to learned simulators that can predict how environments respond to actions, potentially enabling these neural simulators to become both prediction and planning engines.

The team acknowledged that dynamics (motion, change over time) is an important missing piece, though they noted Atlas already contains some dynamic capabilities from training data exposure. They emphasized that exposing models to dynamics during training actually helps static reconstruction by letting the model factor out temporal variation.

Finally, the team proposed that next-view prediction is 'AI complete'—analogous to next-token prediction for language models. Any intelligence task can be framed as predicting the next viewpoint, making this a fundamental primitive for spatial intelligence and potentially general intelligence itself.