How live streaming works: The challenges of low latency video streaming explained | Lex Fridman
A video engineer and entrepreneur discusses the technical challenges of live video streaming, adaptive bitrate algorithms, and ultra-low latency video transmission. The conversation transitions to his new open-source project, Kyber, which targets real-time machine control (robots, drones, remote surgery) with a goal of 4 milliseconds glass-to-glass latency over the internet.
Summary
The conversation begins by distinguishing between offline file playback and live streaming, with the guest noting that modern adaptive streaming is actually less complex than the satellite broadcasting challenges of the late 1990s and early 2000s. Adaptive streaming works by encoding video at multiple resolutions (typically seven) and having the player monitor download speed — if a segment takes more than 50% of its allotted time to download, the player drops to a lower quality tier. The guest notes that the harder problem is deciding when to step quality back up, since frequent quality changes can be psychologically jarring for viewers.
The discussion touches on audio quality degradation being more noticeable than video quality drops, particularly when streaming services switch between full AAC and spectral band replication compressed AAC. The guest observes that viewers are surprisingly tolerant of low video frame rates (e.g., 30fps sports) but immediately detect audio glitches. The complexity of live sports streaming is highlighted — it involves real-time encoding, no time for QA, CDN distribution, DRM protection, and delivery across a wide variety of devices.
The second half of the conversation focuses on Kyber, the guest's new open-source SDK platform for ultra-low latency machine control. Kyber uses a single QUIC (UDP-based) socket to multiplex video, audio, and control inputs (mouse, keyboard, gamepad) while maintaining clock synchronization across multiple sensors. This synchronization is critical for robotics AI training, where data coherence across multiple cameras and sensors must be preserved. The guest explains that existing solutions struggle when scaling beyond one camera.
Kyber uses forward error correction — over-transmitting a small percentage of data — to reconstruct lost packets without the latency penalty of TCP retransmission. A demo at CES showed a 3D-printed rover being controlled from France via a small PCB server. The current achieved latency is 7 milliseconds (Windows-to-Windows), with the encoder consuming ~3.5ms and decoder ~2ms. The ultimate goal is 4 milliseconds glass-to-glass over the internet. Kyber is dual-licensed: AGPL for open-source projects and a commercial license for proprietary products.
Key Insights
- The guest argues that adaptive streaming is fundamentally a CDN congestion problem rather than a video problem, and that the core buffering algorithm is quite basic — if a segment takes more than 50% of its download window to arrive, the player steps down in quality.
- The guest claims that audio quality degradation is more perceptually jarring than video quality drops, specifically noting that switching between full AAC and spectral band replication compressed AAC is immediately noticeable to listeners, while video resolution changes are smoother and less noticed.
- The guest explains that Kyber accounts for clock drift across multiple sensors on a robot, which existing solutions fail to handle beyond a single camera — a critical requirement for training robotics AI models on coherent, time-synchronized multi-sensor data.
- The guest states that Kyber uses forward error correction over a single QUIC (UDP-based) socket, intentionally over-transmitting a few percent of data so that lost packets can be reconstructed client-side without the latency cost of TCP-style retransmission acknowledgment.
- The guest reports that Kyber currently achieves 7 milliseconds glass-to-glass latency (Windows-to-Windows), with approximately 3.5ms consumed by the Nvidia hardware encoder and 2ms by the Intel decoder, leaving very little headroom to reach the 4ms goal without faster codecs or hardware.
Topics
Transcript
[0:02] I would love to sort of uh zoom in and and and talk a little bit more about the distinction between kind of downloading a file and watching it offline versus streaming. So the the complexities, the challenges of streaming. Is there something we could say about what it takes to uh stream files? We've been talking about codecs and I think a lot of that implies encoding and decoding uh without the having to communicate [0:33] over the network. >> Sure. >> Sure. >> Uh so can you can you elaborate like what's required to do over the network stuff? >> Yeah, but it is less complex than it seems compared to everything that we've talked about. Um…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
Is Islam Makhachev the greatest fighter of all time? | Khabib Nurmagomedov and Lex Fridman
Khabib Nurmagomedov discusses Islam Makhachev's potential to become the greatest MMA fighter of all time, comparing him to legendary fighters like GSP, Jon Jones, and Demetrious Johnson. While Khabib believes Islam has the capability to break records and surpass these greats, he expresses concern about the limited time remaining in his career, particularly noting how age affects performance more severely in lighter weight classes.
Why Khabib beat up Lex in training
Khabib explains his top pressure wrestling technique and fighting philosophy, emphasizing the use of Aikido principles to leverage opponent's force against them. He discusses how elite fighters identify and relentlessly pressure their opponent's weaknesses rather than engaging in their strengths.
The secret to Khabib's dominance: I hated to lose | Khabib Nurmagomedov and Lex Fridman
Khabib Nurmagomedov discusses how his hatred of losing, rather than desire to win, drove his dominance in combat sports. He emphasizes the importance of fanatical dedication to training—sometimes 3-5 hours daily across multiple sessions—and mental toughness as prerequisites for reaching the elite level in athletics.
Khabib invites Lex Fridman to Dagestan
Khabib discusses his high-altitude sports training complex in Sildi, Dagestan, built as a continuation of his late father's dream. He explains the physical benefits of mountain training and invites Lex Fridman to visit and film content at the facility, describing the journey from Makhachkala.
Lex Fridman on interviewing Fedor Emelianenko
Lex Fridman recounts a memorable moment watching Fedor Emelianenko at a 2004 Mahachkala fight, where he was impressed by Fedor's meticulous warm-up routine focusing on his face and nose, and his calm post-fight behavior of drinking tea with his team. Lex expresses interest in interviewing Fedor about this experience and asks for help reaching out to the notoriously private fighter.