Video codecs explained: H.264, AV1, HEVC, VVC | Lex Fridman Podcast
This transcript from the Lex Fridman Podcast explains how video codecs work by removing spatial and temporal redundancy in video data. The discussion covers the asymmetric nature of encoding vs. decoding, error resilience requirements, and how modern codecs like AV1 and VVC are actually collections of multiple tools designed for different content types.
Summary
The conversation opens with a definition of video codecs, explaining that their core purpose is to remove redundant data from video streams using mathematical properties. Video contains massive spatial redundancy (e.g., a uniform black background) and temporal redundancy (e.g., a cloud that remains the same across many frames), and codecs exploit both to achieve dramatic compression ratios.
The speakers highlight that compression is computationally asymmetric: encoding is far more expensive than decoding, which makes sense because a file is compressed once but potentially decoded by millions of viewers. This asymmetry is a key design consideration when building a codec.
Error resilience is also discussed as a critical design goal, particularly in the context of UDP network streams (as used historically by VLC). Because UDP drops packets, codecs must be designed so that a decoder can join a stream mid-way and recover gracefully without the full data from the beginning.
The conversation provides an intuitive explanation of video as a grid of RGB pixels repeated 24, 30, or 60 times per second, with the compression target being on the order of 1000x. Redundancy is described as information humans wouldn't notice if missing — such as a repeated cloud or a uniform background color — and codecs exploit this by referencing earlier frames or neighboring pixels instead of storing duplicate data.
Finally, the speakers clarify that modern codecs like AV1, VVC, and the upcoming AV2 are not single monolithic codecs but rather collections of specialized tools. Each tool is optimized for a different type of content — screen sharing, live video, animation, etc. — and the codec dynamically switches between tools depending on the content being compressed, requiring large teams of specialized engineers to develop each component.
Key Insights
- The speakers argue that compression is intentionally asymmetric — encoding uses orders of magnitude more compute than decoding — because a file is compressed once but can be viewed by many, making it economically justified to spend more resources on the encoder side.
- Error resilience is described as a fundamental codec design goal, not an afterthought — codecs must allow decoders to join and begin decoding a stream mid-way, which was a practical necessity for VLC's early use of UDP network feeds that drop packets.
- The speakers state that the target compression ratio for modern video codecs is approximately 1000x, achieved by eliminating data that humans would not notice is missing, such as repeated background elements across frames.
- One speaker explains that increasing memory and compute power directly enables greater compression, because the codec can compare pixels across more frames in the past — but this comes at a significant computational cost, especially at 4K resolution.
- Modern codecs like AV1, AV2, and VVC are described not as single codecs but as collections of multiple coding tools, each optimized for a different content type (screen sharing, video, animation), with the codec dynamically switching tools based on the content being encoded.
Topics
Full transcript available for MurmurCast members
Sign Up to Access