Video codecs explained: H.264, AV1, HEVC, VVC | Lex Fridman Podcast
This transcript from the Lex Fridman Podcast explains how video codecs work by removing spatial and temporal redundancy in video data. The discussion covers the asymmetric nature of encoding vs. decoding, error resilience requirements, and how modern codecs like AV1 and VVC are actually collections of multiple tools designed for different content types.
Summary
The conversation opens with a definition of video codecs, explaining that their core purpose is to remove redundant data from video streams using mathematical properties. Video contains massive spatial redundancy (e.g., a uniform black background) and temporal redundancy (e.g., a cloud that remains the same across many frames), and codecs exploit both to achieve dramatic compression ratios.
The speakers highlight that compression is computationally asymmetric: encoding is far more expensive than decoding, which makes sense because a file is compressed once but potentially decoded by millions of viewers. This asymmetry is a key design consideration when building a codec.
Error resilience is also discussed as a critical design goal, particularly in the context of UDP network streams (as used historically by VLC). Because UDP drops packets, codecs must be designed so that a decoder can join a stream mid-way and recover gracefully without the full data from the beginning.
The conversation provides an intuitive explanation of video as a grid of RGB pixels repeated 24, 30, or 60 times per second, with the compression target being on the order of 1000x. Redundancy is described as information humans wouldn't notice if missing — such as a repeated cloud or a uniform background color — and codecs exploit this by referencing earlier frames or neighboring pixels instead of storing duplicate data.
Finally, the speakers clarify that modern codecs like AV1, VVC, and the upcoming AV2 are not single monolithic codecs but rather collections of specialized tools. Each tool is optimized for a different type of content — screen sharing, live video, animation, etc. — and the codec dynamically switches between tools depending on the content being compressed, requiring large teams of specialized engineers to develop each component.
Key Insights
- The speakers argue that compression is intentionally asymmetric — encoding uses orders of magnitude more compute than decoding — because a file is compressed once but can be viewed by many, making it economically justified to spend more resources on the encoder side.
- Error resilience is described as a fundamental codec design goal, not an afterthought — codecs must allow decoders to join and begin decoding a stream mid-way, which was a practical necessity for VLC's early use of UDP network feeds that drop packets.
- The speakers state that the target compression ratio for modern video codecs is approximately 1000x, achieved by eliminating data that humans would not notice is missing, such as repeated background elements across frames.
- One speaker explains that increasing memory and compute power directly enables greater compression, because the codec can compare pixels across more frames in the past — but this comes at a significant computational cost, especially at 4K resolution.
- Modern codecs like AV1, AV2, and VVC are described not as single codecs but as collections of multiple coding tools, each optimized for a different content type (screen sharing, video, animation), with the codec dynamically switching tools based on the content being encoded.
Topics
Transcript
[0:02] We hinted at awesomeness of codex, the depth and the richness and the complexity of everything involved there. What let's try to define what is a video codec? What what's involved there? What what does it mean to compress something? You already started to hint at it, but can can we elaborate a little bit more? So there's a huge amount of redundancy in any video both spatial and temporal. And the point of any video codec is to remove this redundant data, use mathematical properties as part of this reduction process. So more often than not using several [0:33] orders of magnitude more compute to compress because that's more costly versus both costly both financially and in CPU…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
How the Civil War ended | Gary Gallagher and Lex Fridman
Gary Gallagher and Lex Fridman discuss how the Civil War ended, focusing on Lee's surrender at Appomattox and Grant's magnanimous treatment of Confederate forces. They explore how Grant's generous surrender terms, influenced by Lincoln's reconciliation goals, contrasted with calls from Radical Republicans for harsher punishment and how this approach facilitated national reunification rather than perpetuating resentment.
White supremacy after the Civil War | Gary Gallagher and Lex Fridman
Historian Gary Gallagher discusses Reconstruction, white supremacy, and Confederate memory in post-Civil War America, emphasizing that Northern troops were insufficient to enforce equal rights for freed Black people, and explaining how the "Lost Cause" narrative allowed the South to reframe the war's purpose while Jim Crow became the mechanism for maintaining white supremacy.
The truth about the Civil War - Why people volunteered to fight | Gary Gallagher and Lex Fridman
Historian Gary Gallagher discusses the distinction between causation and motivation in Civil War volunteerism, explaining that while slavery caused the war, most Northern soldiers were motivated by belief in preserving the Union as a democratic republic. He emphasizes that understanding citizen-soldiers' sense of civic duty and American exceptionalism is essential to comprehending why ordinary people risked their lives for the Union.
Christianity and religion during the Civil War | Gary Gallagher and Lex Fridman
Both Union and Confederate sides during the American Civil War were overwhelmingly Protestant Christian and used religious justification for their military efforts. Interestingly, Confederate leaders like Lee and Jackson were notably more overtly religious in their official communications than Union commanders like Grant and Sherman.
Racism in America after slavery was abolished | Gary Gallagher and Lex Fridman
Gary Gallagher discusses how abolishing slavery was only the beginning of achieving equal rights for Black Americans, noting that most Northern whites prioritized saving the Union over ensuring racial equality. He explains that legal changes like the Civil Rights Act took over a century to achieve, while the deeper work of changing societal attitudes and prejudices remains ongoing and vulnerable to exploitation.