Video codecs explained: H.264, AV1, HEVC, VVC | Lex Fridman Podcast
This transcript from the Lex Fridman Podcast explains how video codecs work by removing spatial and temporal redundancy in video data. The discussion covers the asymmetric nature of encoding vs. decoding, error resilience requirements, and how modern codecs like AV1 and VVC are actually collections of multiple tools designed for different content types.
Summary
The conversation opens with a definition of video codecs, explaining that their core purpose is to remove redundant data from video streams using mathematical properties. Video contains massive spatial redundancy (e.g., a uniform black background) and temporal redundancy (e.g., a cloud that remains the same across many frames), and codecs exploit both to achieve dramatic compression ratios.
The speakers highlight that compression is computationally asymmetric: encoding is far more expensive than decoding, which makes sense because a file is compressed once but potentially decoded by millions of viewers. This asymmetry is a key design consideration when building a codec.
Error resilience is also discussed as a critical design goal, particularly in the context of UDP network streams (as used historically by VLC). Because UDP drops packets, codecs must be designed so that a decoder can join a stream mid-way and recover gracefully without the full data from the beginning.
The conversation provides an intuitive explanation of video as a grid of RGB pixels repeated 24, 30, or 60 times per second, with the compression target being on the order of 1000x. Redundancy is described as information humans wouldn't notice if missing — such as a repeated cloud or a uniform background color — and codecs exploit this by referencing earlier frames or neighboring pixels instead of storing duplicate data.
Finally, the speakers clarify that modern codecs like AV1, VVC, and the upcoming AV2 are not single monolithic codecs but rather collections of specialized tools. Each tool is optimized for a different type of content — screen sharing, live video, animation, etc. — and the codec dynamically switches between tools depending on the content being compressed, requiring large teams of specialized engineers to develop each component.
Key Insights
- The speakers argue that compression is intentionally asymmetric — encoding uses orders of magnitude more compute than decoding — because a file is compressed once but can be viewed by many, making it economically justified to spend more resources on the encoder side.
- Error resilience is described as a fundamental codec design goal, not an afterthought — codecs must allow decoders to join and begin decoding a stream mid-way, which was a practical necessity for VLC's early use of UDP network feeds that drop packets.
- The speakers state that the target compression ratio for modern video codecs is approximately 1000x, achieved by eliminating data that humans would not notice is missing, such as repeated background elements across frames.
- One speaker explains that increasing memory and compute power directly enables greater compression, because the codec can compare pixels across more frames in the past — but this comes at a significant computational cost, especially at 4K resolution.
- Modern codecs like AV1, AV2, and VVC are described not as single codecs but as collections of multiple coding tools, each optimized for a different content type (screen sharing, video, animation), with the codec dynamically switching tools based on the content being encoded.
Topics
Transcript
[0:02] We hinted at awesomeness of codex, the depth and the richness and the complexity of everything involved there. What let's try to define what is a video codec? What what's involved there? What what does it mean to compress something? You already started to hint at it, but can can we elaborate a little bit more? So there's a huge amount of redundancy in any video both spatial and temporal. And the point of any video codec is to remove this redundant data, use mathematical properties as part of this reduction process. So more often than not using several [0:33] orders of magnitude more compute to compress because that's more costly versus both costly both financially and in CPU…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
Anti-matter & nuclear weapons: Why technology is always a double-edge sword | Don Lincoln
Don Lincoln discusses how advanced energy sources like nuclear fusion, fission, and antimatter represent transformative but double-edged technologies. He argues that science's role is to understand nature, while society must collectively decide how to apply that knowledge. The conversation concludes with a celebration of humanity's innate curiosity as the driver of civilizational progress.
Why antimatter costs $63 trillion dollars to produce | Don Lincoln and Lex Fridman
Don Lincoln and Lex Fridman discuss the extreme difficulty and cost of producing antimatter, noting that Fermilab could only produce about one nanogram per year. They explore the theoretical potential of antimatter as a propulsion system for space travel, while emphasizing that the core challenge is an engineering problem of concentrating energy, not a physics breakthrough.
Is loop quantum gravity wrong? - physicist explains | Don Lincoln and Lex Fridman
Don Lincoln explains the differences between loop quantum gravity and string theory, noting that loop quantum gravity attempts to quantize space itself rather than unify all forces. He discusses how an early prediction of loop quantum gravity — that light speed would vary by frequency — was disproven by gamma ray burst observations, but the theory adapted. He also highlights the landmark gravitational wave observation confirming that gravity travels at the speed of light.
Can antimatter be used as rocket fuel? | Don Lincoln and Lex Fridman
Don Lincoln and Lex Fridman discuss the feasibility of using antimatter as rocket fuel, noting that while it is physically possible, the cost of production (estimated at $62-63 trillion per gram) and containment challenges make it impractical. Lincoln explains that antimatter propulsion is fundamentally an engineering problem rather than a physics mystery, and that breakthroughs would likely come from finding new ways to concentrate energy rather than new physics theory.
Speed of light explained: Was Einstein's theory correct? | Don Lincoln and Lex Fridman
Don Lincoln explains Einstein's special relativity, focusing on the two core premises: the universality of natural laws and the constant speed of light for all observers. He describes modern particle physics experiments that have empirically confirmed Einstein's conjecture. He also reflects on how understanding space-time makes the concept of a universal speed limit intuitive rather than bizarre.