H.264 codec vs MP4 container: Difference explained | Lex Fridman Podcast
This segment from the Lex Fridman Podcast explains the technical distinction between video containers (like MP4 and MKV) and video codecs (like H.264 and AV1). The discussion covers how VLC was designed to handle broken or mislabeled video files by not trusting file extensions. It also explores how video codecs achieve massive compression ratios by exploiting spatial and temporal redundancy in video data.
Summary
The conversation begins by distinguishing between video containers and codecs. Containers (also called muxers) are file formats that hold multiple tracks — video, audio, subtitles — together. Common containers include MP4, MKV (Matroska), AVI, and MOV. Codecs, on the other hand, are coder-decoder systems responsible for compressing and decompressing the actual media data. A frequent source of public confusion is conflating MP4 (a container) with H.264 (a codec), though the speakers note this confusion is partly the industry's fault, since H.264 is formally known as MPEG-4 Part 10, which shares naming with the MP4 container.
The discussion then turns to how tools like VLC and FFmpeg handle real-world video files, which are often mislabeled or malformed. Rather than trusting file extensions, both tools probe the file contents directly to determine the actual container and codec. This philosophy of 'not trusting inputs' originated from VLC's roots as a client for UDP-based video streaming, where packet loss and data corruption were common. This also made VLC popular during early internet piracy days, when partially downloaded AVI files (whose metadata sits at the end) could still be played.
The conversation then dives into what video codecs actually do: they remove spatial redundancy (e.g., identical pixels within a single frame) and temporal redundancy (e.g., unchanged background elements across multiple frames) to achieve compression ratios of 100x to 1000x. More compute-intensive encoding allows for better compression, while decoding is generally less expensive — an asymmetric trade-off that makes sense since content is encoded once but potentially decoded by millions of viewers.
Finally, the speakers discuss modern codec design, noting that codecs like AV1, AV2, and VVC are not single algorithms but collections of tools tailored to different content types — screen sharing, live video, animation, etc. These adaptive tool sets allow the codec to dynamically switch compression strategies based on the content being encoded, such as switching from a PowerPoint slide to a video mid-stream on a platform like Zoom.
Key Insights
- The speaker explains that confusing MP4 with H.264 is understandable because H.264 is formally named 'MPEG-4 Part 10,' making it part of the same MPEG-4 specification umbrella as the MP4 container — so the industry itself created the naming confusion.
- VLC's resilience to broken or mislabeled video files stems from its origins as a client for UDP-based streaming, where packet loss was common — leading to a foundational engineering philosophy of never trusting inputs.
- VLC became popular during early internet piracy because AVI files store their metadata at the end of the file, meaning partially downloaded files were unplayable by most software — but VLC would attempt to interpret them anyway.
- The speakers argue that compression is asymmetric in cost: encoding is far more compute-intensive than decoding, which makes sense because content is compressed once but potentially decoded by a large number of viewers.
- Modern codecs like AV1 and VVC are not single algorithms but collections of tools designed to handle different content types — such as screen sharing, animation, and live video — and dynamically switch strategies based on what is being encoded.
Topics
Transcript
[0:02] We mentioned codecs, but I don't think you mentioned containers. So, what what's the actual containers for some of the stuff we're talking about? So, people are familiar with MP4 uh MKV. So, anyway, what what are containers versus uh the thing that goes inside? So, the container is what we call also the moxer, right? When I say demoxing, it means decontenizing, right? So actually if you look MX mean multiplexer and de multiplexer right MX [0:35] and demox are those and same codec is actually coder decoder right um and um so containers are this collection of multiple tracks right so it's a what normal people call the file format but it's a bit more um subtle…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
Khabib vs Conor McGregor fight: Khabib on why it was the biggest fight of all time
Khabib Nurmagomedov reflects on his historic fight with Conor McGregor, emphasizing the significance of their rivalry and the psychological aspect of fighting. He believes that their matchup transcended MMA, impacting various sports and that such monumental fights could happen again in the future.
Khabib on Joe Rogan podcast | Khabib Nurmagomedov and Lex Fridman
Khabib Nurmagomedov expresses anticipation for a future conversation with Joe Rogan, appreciating his contributions to the UFC and podcasting. He also discusses the importance of language, specifically his desire to improve his Arabic skills while acknowledging the differences between literary and conversational Arabic.
Khabib on greatness of Islam Makhachev | Khabib Nurmagomedov and Lex Fridman
Khabib Nurmagomedov discusses the attributes that make Islam Makhachev an exceptional fighter, highlighting their shared history, discipline, and the sacrifices required to reach the top levels of MMA. He also reflects on the generational cycle of fighters in their gym and contemplates Islam's potential to become one of the greatest MMA fighters of all time.
How the Civil War ended | Gary Gallagher and Lex Fridman
Gary Gallagher and Lex Fridman discuss how the Civil War ended, focusing on Lee's surrender at Appomattox and Grant's magnanimous treatment of Confederate forces. They explore how Grant's generous surrender terms, influenced by Lincoln's reconciliation goals, contrasted with calls from Radical Republicans for harsher punishment and how this approach facilitated national reunification rather than perpetuating resentment.
White supremacy after the Civil War | Gary Gallagher and Lex Fridman
Historian Gary Gallagher discusses Reconstruction, white supremacy, and Confederate memory in post-Civil War America, emphasizing that Northern troops were insufficient to enforce equal rights for freed Black people, and explaining how the "Lost Cause" narrative allowed the South to reframe the war's purpose while Jim Crow became the mechanism for maintaining white supremacy.