TechnicalOpinion

Vertical Foundation Models

Y Combinator

The speaker advocates for vertical foundation models—specialized AI models built for specific domains like manufacturing, energy, and biology—that can match or exceed general-purpose models while requiring significantly less compute. These domain-specific models can learn new tasks with zero or few examples, with Google's Times FM3 for time series forecasting cited as a successful example.

Summary

The speaker discusses the value of specialized foundation models as an alternative to general-purpose AI. They argue that vertical foundation models offer developers a type decision and probability output without requiring a trained classifier for each new task. The key advantage is computational efficiency: these specialized models should match or beat general-purpose models while consuming orders of magnitude less inference compute, making them viable for deployment across diverse hardware. Google's Times FM3 is highlighted as a concrete example—a 330 million parameter model that forecasts related time series simultaneously without task-specific fine-tuning, outputting complete forecasts in a single forward pass. The model training happens once, with users providing their own historical data as context. The speaker identifies the primary challenge as obtaining high-quality, domain-specific private data. They suggest that developers with deep vertical domain expertise could build compute-efficient, domain-specific foundation models capable of zero-shot or few-shot learning for new tasks.

Key Insights

  • Vertical foundation models eliminate the need for developers to train a separate classifier for every new task by providing type decisions and probabilities directly
  • The success metric for vertical foundation models is matching or beating general-purpose models while requiring orders of magnitude less inference compute to make deployment economically feasible
  • Google's Times FM3 demonstrates that a 330 million parameter model can forecast multiple related time series simultaneously without task-specific fine-tuning and complete inference in one forward pass
  • Vertical foundation models require training only once, with each new user providing their own historical context rather than requiring model retraining
  • Obtaining high-quality, domain-specific private data is identified as the primary technical challenge in building effective vertical foundation models

Topics

Vertical foundation modelsDomain-specific AICompute efficiencyZero-shot and few-shot learningTime series forecastingGeneral-purpose vs specialized models

Transcript

[0:00] I think Jeff hit a nerve because it gives developers a type decision and a probability [music] instead of generated text without training a classifier for every new task. I'd like to see more research labs build specialized foundation models for manufacturing, energy, and biology that work zero-shot or from a handful of examples. The bar is matching or beating general-purpose models at orders of magnitude less inference compute, so they're cheap enough to run on everything. For example, Google's Times FM3 shows what [0:31] this looks like for forecasting. It is a 330 million parameter model that forecasts related time series at once with no task-specific fine-tuning and outputs the whole forecast in one forward pass. >> [music]…

Full transcript available for MurmurCast members

Sign Up to Access

More from Y Combinator

Get AI summaries like this delivered to your inbox daily

Get AI summaries delivered to your inbox

MurmurCast summarizes your YouTube channels, podcasts, and newsletters into one daily email digest.