H.264 codec vs MP4 container: Difference explained | Lex Fridman Podcast
This segment from the Lex Fridman Podcast explains the technical distinction between video containers (like MP4 and MKV) and video codecs (like H.264 and AV1). The discussion covers how VLC was designed to handle broken or mislabeled video files by not trusting file extensions. It also explores how video codecs achieve massive compression ratios by exploiting spatial and temporal redundancy in video data.
Summary
The conversation begins by distinguishing between video containers and codecs. Containers (also called muxers) are file formats that hold multiple tracks — video, audio, subtitles — together. Common containers include MP4, MKV (Matroska), AVI, and MOV. Codecs, on the other hand, are coder-decoder systems responsible for compressing and decompressing the actual media data. A frequent source of public confusion is conflating MP4 (a container) with H.264 (a codec), though the speakers note this confusion is partly the industry's fault, since H.264 is formally known as MPEG-4 Part 10, which shares naming with the MP4 container.
The discussion then turns to how tools like VLC and FFmpeg handle real-world video files, which are often mislabeled or malformed. Rather than trusting file extensions, both tools probe the file contents directly to determine the actual container and codec. This philosophy of 'not trusting inputs' originated from VLC's roots as a client for UDP-based video streaming, where packet loss and data corruption were common. This also made VLC popular during early internet piracy days, when partially downloaded AVI files (whose metadata sits at the end) could still be played.
The conversation then dives into what video codecs actually do: they remove spatial redundancy (e.g., identical pixels within a single frame) and temporal redundancy (e.g., unchanged background elements across multiple frames) to achieve compression ratios of 100x to 1000x. More compute-intensive encoding allows for better compression, while decoding is generally less expensive — an asymmetric trade-off that makes sense since content is encoded once but potentially decoded by millions of viewers.
Finally, the speakers discuss modern codec design, noting that codecs like AV1, AV2, and VVC are not single algorithms but collections of tools tailored to different content types — screen sharing, live video, animation, etc. These adaptive tool sets allow the codec to dynamically switch compression strategies based on the content being encoded, such as switching from a PowerPoint slide to a video mid-stream on a platform like Zoom.
Key Insights
- The speaker explains that confusing MP4 with H.264 is understandable because H.264 is formally named 'MPEG-4 Part 10,' making it part of the same MPEG-4 specification umbrella as the MP4 container — so the industry itself created the naming confusion.
- VLC's resilience to broken or mislabeled video files stems from its origins as a client for UDP-based streaming, where packet loss was common — leading to a foundational engineering philosophy of never trusting inputs.
- VLC became popular during early internet piracy because AVI files store their metadata at the end of the file, meaning partially downloaded files were unplayable by most software — but VLC would attempt to interpret them anyway.
- The speakers argue that compression is asymmetric in cost: encoding is far more compute-intensive than decoding, which makes sense because content is compressed once but potentially decoded by a large number of viewers.
- Modern codecs like AV1 and VVC are not single algorithms but collections of tools designed to handle different content types — such as screen sharing, animation, and live video — and dynamically switch strategies based on what is being encoded.
Topics
Full transcript available for MurmurCast members
Sign Up to Access