Future of holograms, brain signal playback, smell, and touch in multimedia software | Lex Fridman
Developers of VLC and FFmpeg discuss the future of multimedia frameworks, predicting expansion into holograms, brain-computer interfaces, smell, touch, and volumetric video. They argue that multimedia should be redefined as any timed data representing human senses. They also critique Dolby for shifting from innovation to patent licensing.
Summary
The conversation centers on the long-term trajectory of FFmpeg and VLC, two foundational open-source multimedia frameworks. The speakers express high confidence that FFmpeg will exist in 100 years, while VLC's survival is considered more uncertain. Looking 5–20 years ahead, they anticipate that multimedia will expand far beyond audio and video to encompass brain-computer interface data (referencing Neuralink), volumetric video, point clouds, RGBD depth data, and XR/VR streaming.
The speakers redefine multimedia as 'a digital representation of several streams for the human senses,' arguing this naturally includes smell and touch. They describe existing work on haptic plugins in VLC used for 4D cinema experiences with hydraulic motion platforms, and they envision future codec tracks for stereo smell and haptic feedback. They frame FFmpeg and VLC not as contributors but as maintainers who shape architecture so that external contributors can add new format modules over time.
The speakers describe a recurring industry pattern in emerging multimedia formats: initial fragmentation into multiple competing standards, followed by a hype decline, consolidation around a community-driven standard, and eventual adoption by market leaders under pressure. They cite 3D audio from six or seven years ago as a cautionary example of premature hype followed by a more mature revival.
Finally, the speakers criticize Dolby, characterizing it as a once-innovative company that defined the standard for sound but has since shifted its focus toward lawyers and licensing revenue rather than engineering innovation, comparing its trajectory unfavorably to HP.
Key Insights
- The speakers argue that multimedia should be defined as 'a digital representation of several streams for the human senses,' which inherently includes smell, touch, and eventually brain wave data — making expansion into those domains logically inevitable for FFmpeg and VLC.
- VLC already has a haptic plugin, not included in the standard release, that transports physical and movement data synchronized with content, used in 4D cinema theme park experiences with hydraulic arms.
- The speakers describe their role as maintainers rather than contributors — their job is to architect the frameworks so that a large community (around 150 VLC contributors and 300 FFmpeg contributors per year) can plug in new modules for emerging formats.
- The speakers identify a consistent pattern in multimedia standardization: new formats start with five competing standards driven by hype, the hype collapses, secondary players converge on a shared standard, it gets implemented, and then the market leader is eventually pressured to adopt it — citing 3D audio from six or seven years ago as an example.
- The speakers characterize Dolby as a company that once defined what sound was with great engineering, but has shifted to being primarily driven by lawyers and licensing, implying it no longer innovates at the level it once did.
Topics
Full transcript available for MurmurCast members
Sign Up to Access