Video compression explained: H.264, AV1, ProRes | Lex Fridman Podcast
This podcast transcript covers video compression fundamentals, explaining codec and container combinations, IPB frame types, and the complexity of compression parameters. The speakers discuss how codecs like H.264, AV1, and ProRes serve different use cases, and highlight the remarkable engineering behind modern video streaming at scale.
Summary
The conversation begins by outlining the most common codec-container combinations in modern video: H.264 in MP4, AV1 in MP4/WebM, and ProRes in its own containers. ProRes is explained as Apple's editing-focused codec, originally designed for Final Cut Pro, optimized for fast decoding and seeking rather than distribution efficiency. Its key characteristic is being an 'intra-only' codec with no temporal compression, meaning every frame is self-contained.
The speakers then dive into IPB frame types. I-frames (keyframes) are complete, self-contained images similar to JPEGs. P-frames (predicted frames) reference previous frames and only store the differences, requiring access to a prior I-frame to decode. B-frames (bi-directionally predicted frames) are the most complex, capable of referencing both past and future frames, meaning the decoding order differs from the display order — a concept the speakers describe as 'mind-blowing.'
The concept of intra-refresh is introduced as an alternative to traditional I-frames, used in streaming platforms like Kyber, where an I-frame is gradually built up across the stream rather than sent as a discrete complete frame. The default GOP (Group of Pictures) size in FFmpeg for H.264 is noted as around 250 frames.
The discussion then turns to the enormous number of tunable parameters in video encoding — resolution, frame rate, codec choice, bitrate mode (constant bitrate vs. constant quality), QP values, GOP length, and B-frame counts. The speakers note that thousands of professionals at companies like YouTube, Netflix, and Meta are dedicated solely to optimizing these parameters for different content types and delivery scenarios.
Finally, YouTube is highlighted as a uniquely complex engineering challenge, requiring both upload encoding at massive scale and delivery across all resolutions with minimal latency — even for videos with very few viewers. The conversation closes with a historical anecdote about Google Video using VLC via ActiveX in Internet Explorer, contrasted with today's approach of compiling VLC and FFmpeg to WebAssembly to run inside JavaScript virtual machines in the browser.
Key Insights
- ProRes is an intra-only codec with no temporal compression, meaning every frame is a complete image, which makes it ideal for editing workflows where fast seeking and cutting are required — a fundamentally different use case than distribution codecs.
- B-frames can depend on frames that appear in the future in the stream, meaning the decoding order is not the same as the display order — the decoder must buffer and decode a future frame before it can decode the current B-frame.
- Intra-refresh, used in platforms like Kyber, eliminates discrete I-frames entirely by gradually building up a complete frame across the continuous stream, refreshing certain parts of the image over time.
- The speakers argue that thousands of employees at companies like YouTube, Netflix, and Meta are not writing codecs but are instead solely dedicated to finding the right encoding parameters for specific content types and delivery formats.
- The fact that decoders from different manufacturers across the world — one in the US, one elsewhere — can decode the same stream and produce bit-for-bit identical output is described as 'quite remarkable,' especially as codecs grow increasingly complex.
Topics
Transcript
[0:03] We can lay out let's say the top five codec container combos would be H264 inside MP4 containers AV1 inside MP4 webm containers uh ProRes for nonlinear editing uh inside containers. So for people who don't know, I guess ProRes is >> it's Apple's codec for editing originally for Final Cut Pro and it's designed to be fast to decode, fast to [0:34] seek because an editor will need to move very quickly. So it's a different use case to the distribution element. >> There's no or very minimal uh temporal compression. >> There's none here. There's none in so you can cut. So you can do cuts. >> This is what we call intra only codeex. Right? So…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
Anti-matter & nuclear weapons: Why technology is always a double-edge sword | Don Lincoln
Don Lincoln discusses how advanced energy sources like nuclear fusion, fission, and antimatter represent transformative but double-edged technologies. He argues that science's role is to understand nature, while society must collectively decide how to apply that knowledge. The conversation concludes with a celebration of humanity's innate curiosity as the driver of civilizational progress.
Why antimatter costs $63 trillion dollars to produce | Don Lincoln and Lex Fridman
Don Lincoln and Lex Fridman discuss the extreme difficulty and cost of producing antimatter, noting that Fermilab could only produce about one nanogram per year. They explore the theoretical potential of antimatter as a propulsion system for space travel, while emphasizing that the core challenge is an engineering problem of concentrating energy, not a physics breakthrough.
Is loop quantum gravity wrong? - physicist explains | Don Lincoln and Lex Fridman
Don Lincoln explains the differences between loop quantum gravity and string theory, noting that loop quantum gravity attempts to quantize space itself rather than unify all forces. He discusses how an early prediction of loop quantum gravity — that light speed would vary by frequency — was disproven by gamma ray burst observations, but the theory adapted. He also highlights the landmark gravitational wave observation confirming that gravity travels at the speed of light.
Can antimatter be used as rocket fuel? | Don Lincoln and Lex Fridman
Don Lincoln and Lex Fridman discuss the feasibility of using antimatter as rocket fuel, noting that while it is physically possible, the cost of production (estimated at $62-63 trillion per gram) and containment challenges make it impractical. Lincoln explains that antimatter propulsion is fundamentally an engineering problem rather than a physics mystery, and that breakthroughs would likely come from finding new ways to concentrate energy rather than new physics theory.
Speed of light explained: Was Einstein's theory correct? | Don Lincoln and Lex Fridman
Don Lincoln explains Einstein's special relativity, focusing on the two core premises: the universality of natural laws and the constant speed of light for all observers. He describes modern particle physics experiments that have empirically confirmed Einstein's conjecture. He also reflects on how understanding space-time makes the concept of a universal speed limit intuitive rather than bizarre.