The impossible task of testing FFmpeg code | Lex Fridman Podcast
A developer discusses FFmpeg's FATE (FFmpeg Automated Testing Environment) system, which uses volunteer-run machines to test an enormous matrix of compiler, OS, and architecture combinations. The conversation also touches on the speaker's company, Open Broadcast, which builds sports broadcasting equipment and deals with complex 10-bit video format conversion challenges. The scale of testing configurations is described as beyond a simple matrix — more like a multi-dimensional pivot table.
Summary
The conversation opens with a discussion of FFmpeg's automated testing system called FATE (FFmpeg Automated Testing Environment). Because FFmpeg runs across a vast number of operating systems and can be compiled with many different compilers, the number of test configurations is enormous. The speaker walks through the fate.ffmpeg.org website, showing combinations of compilers (e.g., Apple Clang, GCC, Intel compiler, Visual Studio), operating systems (Mac OS, iOS, tvOS), and instruction set architectures (PowerPC, RISC, x86). All of these test systems are run by volunteers, including machines the speaker himself hosts in his office.
The speaker explains that FATE is critical because complex C code in FFmpeg can sometimes be miscompiled — meaning the compiler itself produces incorrect output. This is particularly impactful in video processing because frames have dependencies on each other, so even a small error can cascade into significant visual glitches. FATE allows developers to catch issues that may not appear locally but break specific compiler versions on specific platforms.
The conversation then shifts to the speaker's day job at his company, Open Broadcast (clarified as unrelated to the free OBS streaming software). The company builds equipment for broadcasting sports matches between TV stadiums and studios. A major technical challenge they face is handling 10-bit video, which cannot be processed natively on a CPU in its native format and must be stored in 16-bit containers, wasting 6 bits. Various packing formats exist to use that space more efficiently, which matters greatly for network and PCI Express bus bandwidth. The speaker mentions they maintain roughly a 5x6 or 6x6 matrix of format-to-format conversions, all written in handwritten assembly and supporting different CPU generations — a process he describes as 'really traumatic.'
Key Insights
- The speaker explains that all FATE test systems are run by volunteers, including machines he personally hosts in his office, meaning the breadth of FFmpeg's cross-platform testing relies entirely on community participation.
- The speaker argues that FATE is essential because compilers can sometimes miscompile valid C code, and in video processing this is especially dangerous since frame dependencies mean even a small output error can cascade into large visible glitches.
- The speaker describes the FFmpeg test configuration space not as a simple matrix but as a 'pivot table of different combinations,' highlighting the combinatorial explosion of OS, compiler, and architecture variants.
- The speaker explains that 10-bit video cannot be processed natively on a CPU and must be packed into 16-bit storage, wasting 6 bits, which creates significant bandwidth pressure on interfaces like PCI Express — driving the need for specialized packing formats.
- The speaker reveals that his company maintains a roughly 5x5 or 6x6 matrix of every format-to-every-other-format conversion for 10-bit video, all written in handwritten assembly with support for multiple CPU generations, which he describes as 'really traumatic.'
Topics
Full transcript available for MurmurCast members
Sign Up to Access