How live streaming works: The challenges of low latency video streaming explained | Lex Fridman
A video engineer and entrepreneur discusses the technical challenges of live video streaming, adaptive bitrate algorithms, and ultra-low latency video transmission. The conversation transitions to his new open-source project, Kyber, which targets real-time machine control (robots, drones, remote surgery) with a goal of 4 milliseconds glass-to-glass latency over the internet.
Summary
The conversation begins by distinguishing between offline file playback and live streaming, with the guest noting that modern adaptive streaming is actually less complex than the satellite broadcasting challenges of the late 1990s and early 2000s. Adaptive streaming works by encoding video at multiple resolutions (typically seven) and having the player monitor download speed — if a segment takes more than 50% of its allotted time to download, the player drops to a lower quality tier. The guest notes that the harder problem is deciding when to step quality back up, since frequent quality changes can be psychologically jarring for viewers.
The discussion touches on audio quality degradation being more noticeable than video quality drops, particularly when streaming services switch between full AAC and spectral band replication compressed AAC. The guest observes that viewers are surprisingly tolerant of low video frame rates (e.g., 30fps sports) but immediately detect audio glitches. The complexity of live sports streaming is highlighted — it involves real-time encoding, no time for QA, CDN distribution, DRM protection, and delivery across a wide variety of devices.
The second half of the conversation focuses on Kyber, the guest's new open-source SDK platform for ultra-low latency machine control. Kyber uses a single QUIC (UDP-based) socket to multiplex video, audio, and control inputs (mouse, keyboard, gamepad) while maintaining clock synchronization across multiple sensors. This synchronization is critical for robotics AI training, where data coherence across multiple cameras and sensors must be preserved. The guest explains that existing solutions struggle when scaling beyond one camera.
Kyber uses forward error correction — over-transmitting a small percentage of data — to reconstruct lost packets without the latency penalty of TCP retransmission. A demo at CES showed a 3D-printed rover being controlled from France via a small PCB server. The current achieved latency is 7 milliseconds (Windows-to-Windows), with the encoder consuming ~3.5ms and decoder ~2ms. The ultimate goal is 4 milliseconds glass-to-glass over the internet. Kyber is dual-licensed: AGPL for open-source projects and a commercial license for proprietary products.
Key Insights
- The guest argues that adaptive streaming is fundamentally a CDN congestion problem rather than a video problem, and that the core buffering algorithm is quite basic — if a segment takes more than 50% of its download window to arrive, the player steps down in quality.
- The guest claims that audio quality degradation is more perceptually jarring than video quality drops, specifically noting that switching between full AAC and spectral band replication compressed AAC is immediately noticeable to listeners, while video resolution changes are smoother and less noticed.
- The guest explains that Kyber accounts for clock drift across multiple sensors on a robot, which existing solutions fail to handle beyond a single camera — a critical requirement for training robotics AI models on coherent, time-synchronized multi-sensor data.
- The guest states that Kyber uses forward error correction over a single QUIC (UDP-based) socket, intentionally over-transmitting a few percent of data so that lost packets can be reconstructed client-side without the latency cost of TCP-style retransmission acknowledgment.
- The guest reports that Kyber currently achieves 7 milliseconds glass-to-glass latency (Windows-to-Windows), with approximately 3.5ms consumed by the Nvidia hardware encoder and 2ms by the Intel decoder, leaving very little headroom to reach the 4ms goal without faster codecs or hardware.
Topics
Full transcript available for MurmurCast members
Sign Up to Access