Video codecs explained: H.264, AV1, HEVC, VVC | Lex Fridman Podcast
This transcript from the Lex Fridman Podcast explains how video codecs work by removing spatial and temporal redundancy in video data. The discussion covers the asymmetric nature of encoding vs. decoding, error resilience requirements, and how modern codecs like AV1 and VVC are actually collections of multiple tools designed for different content types.
Summary
The conversation opens with a definition of video codecs, explaining that their core purpose is to remove redundant data from video streams using mathematical properties. Video contains massive spatial redundancy (e.g., a uniform black background) and temporal redundancy (e.g., a cloud that remains the same across many frames), and codecs exploit both to achieve dramatic compression ratios.
The speakers highlight that compression is computationally asymmetric: encoding is far more expensive than decoding, which makes sense because a file is compressed once but potentially decoded by millions of viewers. This asymmetry is a key design consideration when building a codec.
Error resilience is also discussed as a critical design goal, particularly in the context of UDP network streams (as used historically by VLC). Because UDP drops packets, codecs must be designed so that a decoder can join a stream mid-way and recover gracefully without the full data from the beginning.
The conversation provides an intuitive explanation of video as a grid of RGB pixels repeated 24, 30, or 60 times per second, with the compression target being on the order of 1000x. Redundancy is described as information humans wouldn't notice if missing — such as a repeated cloud or a uniform background color — and codecs exploit this by referencing earlier frames or neighboring pixels instead of storing duplicate data.
Finally, the speakers clarify that modern codecs like AV1, VVC, and the upcoming AV2 are not single monolithic codecs but rather collections of specialized tools. Each tool is optimized for a different type of content — screen sharing, live video, animation, etc. — and the codec dynamically switches between tools depending on the content being compressed, requiring large teams of specialized engineers to develop each component.
Key Insights
- The speakers argue that compression is intentionally asymmetric — encoding uses orders of magnitude more compute than decoding — because a file is compressed once but can be viewed by many, making it economically justified to spend more resources on the encoder side.
- Error resilience is described as a fundamental codec design goal, not an afterthought — codecs must allow decoders to join and begin decoding a stream mid-way, which was a practical necessity for VLC's early use of UDP network feeds that drop packets.
- The speakers state that the target compression ratio for modern video codecs is approximately 1000x, achieved by eliminating data that humans would not notice is missing, such as repeated background elements across frames.
- One speaker explains that increasing memory and compute power directly enables greater compression, because the codec can compare pixels across more frames in the past — but this comes at a significant computational cost, especially at 4K resolution.
- Modern codecs like AV1, AV2, and VVC are described not as single codecs but as collections of multiple coding tools, each optimized for a different content type (screen sharing, video, animation), with the codec dynamically switching tools based on the content being encoded.
Topics
Transcript
[0:02] We hinted at awesomeness of codex, the depth and the richness and the complexity of everything involved there. What let's try to define what is a video codec? What what's involved there? What what does it mean to compress something? You already started to hint at it, but can can we elaborate a little bit more? So there's a huge amount of redundancy in any video both spatial and temporal. And the point of any video codec is to remove this redundant data, use mathematical properties as part of this reduction process. So more often than not using several [0:33] orders of magnitude more compute to compress because that's more costly versus both costly both financially and in CPU…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
Surprising origin of antipsychotic drugs | Andrew Scull and Lex Fridman
Andrew Scull explains how chlorpromazine, an antihistamine synthesized in the 1880s, was accidentally discovered to have psychiatric applications in the 1950s through French Navy lieutenant Alain Laborit's experiments. The drug's rapid adoption was driven by pharmaceutical companies' aggressive marketing to politicians and hospital administrators rather than by psychiatrists' initial enthusiasm, transforming it from a 'major tranquilizer' into the first 'antipsychotic' drug.
Cognitive Behavioral Therapy (CBT) vs Psychoanalysis | Andrew Scull and Lex Fridman
Andrew Scull discusses how clinical psychology and cognitive behavioral therapy (CBT) emerged after WWII as alternatives to psychoanalysis, becoming dominant through better empirical evidence, shorter treatment duration, and federal funding advantages. CBT's symptom-focused approach proved more measurable and reproducible than psychoanalytic theory, though its effectiveness remains limited for serious mental disorders.
Do anti-depressant drugs work? | Andrew Scull and Lex Fridman
Andrew Scull discusses the history and efficacy of antidepressants, particularly SSRIs like Prozac, explaining that while they statistically outperform placebo, the clinical improvement is often marginal and comes with significant side effects including sexual dysfunction and withdrawal difficulties. He also addresses 'diagnostic creep' in psychiatry, where conditions become increasingly broadly defined and diagnosed over time.
How drugs and psycho-pharmacology changed mental health | Andrew Scull and Lex Fridman
Andrew Scull traces the accidental discovery of psychopharmacology, beginning with chlorpromazine's unexpected psychiatric effects in the 1950s, and discusses how pharmaceutical companies transformed mental health treatment while also introducing serious side effects like tardive dyskinesia and metabolic complications that remain inadequately addressed.
Freud's radical theory of the unconscious mind | Andrew Scull and Lex Fridman
Andrew Scull discusses Freud's development of psychoanalytic theory, including his structural model of the mind (id, ego, superego) and his theoretical innovations like dream interpretation and Freudian slips. The conversation explores how Freudian ideas gained popularity among intellectuals and artists in the 1920s-30s, particularly following WWI's shell shock crisis, despite remaining largely outside mainstream psychiatric institutions that dealt with severely disturbed patients.