The insane difficulty of reverse engineering video codecs | Lex Fridman Podcast

The conversation begins with a historical overview of reverse engineering efforts in the video codec space, starting with proprietary formats like Windows Media and RealMedia in the 2000s, and progressing into the more complex era of the 2010s. A central figure discussed is Costa, a Ukrainian developer living in Germany, described as a borderline genius who reverse engineered binary blobs of 20-30 megabytes — a scale far beyond typical work, where even a 1 megabyte binary is estimated to take about a month of work.

The GoToMeeting codec is used as a case study. The speakers explain the practical problem: recordings made in GoToMeeting used a proprietary codec, and without the original player or decoder, the video could not be played. The reverse engineering process involved finding the relevant software module, hooking into it to extract raw YUV data as a reference, then opening a disassembler and using pattern recognition and intuition to identify structures like DCT transforms and entropy coding — all without documentation, treating the binary itself as the specification.

The discussion dives into the technical challenges: working entirely in a black box for long periods, debugging purely in memory, stepping through millions of instructions at the CPU level, sometimes needing to pause a virtual machine to dump memory due to DRM or encryption layers. A comparison is made to archaeology — reconstructing an entire human from tiny fragments with a small brush.

The concept of bit exactness is explained: from roughly the 2000s onward, most video codecs require that every compliant decoder produce exactly the same output bits for a given input. This was notably absent in MPEG-2 in the 1990s, acknowledged as one of the biggest mistakes of that era, with Yuri Resnik cited as someone who has publicly acknowledged this. The conversation also touches on the importance of sample collection — finding test files is critical, and sometimes a lucky find of thousands of files from a company that depended on the format enables thorough testing.

The segment closes with discussion of FFmpeg's unique position as a near-monopoly in open source multimedia decoding, where each new codec added increases the value of the entire library. Yuri Resnik is highlighted as a highly respected figure in the multimedia standards community, and the speakers reflect warmly on the community of reverse engineers and assembly-level developers who made much of this work possible.