How live streaming works: The challenges of low latency video streaming explained | Lex Fridman
A video engineer and entrepreneur discusses the technical challenges of live video streaming, adaptive bitrate algorithms, and ultra-low latency video transmission. The conversation transitions to his new open-source project, Kyber, which targets real-time machine control (robots, drones, remote surgery) with a goal of 4 milliseconds glass-to-glass latency over the internet.
Summary
The conversation begins by distinguishing between offline file playback and live streaming, with the guest noting that modern adaptive streaming is actually less complex than the satellite broadcasting challenges of the late 1990s and early 2000s. Adaptive streaming works by encoding video at multiple resolutions (typically seven) and having the player monitor download speed — if a segment takes more than 50% of its allotted time to download, the player drops to a lower quality tier. The guest notes that the harder problem is deciding when to step quality back up, since frequent quality changes can be psychologically jarring for viewers.
The discussion touches on audio quality degradation being more noticeable than video quality drops, particularly when streaming services switch between full AAC and spectral band replication compressed AAC. The guest observes that viewers are surprisingly tolerant of low video frame rates (e.g., 30fps sports) but immediately detect audio glitches. The complexity of live sports streaming is highlighted — it involves real-time encoding, no time for QA, CDN distribution, DRM protection, and delivery across a wide variety of devices.
The second half of the conversation focuses on Kyber, the guest's new open-source SDK platform for ultra-low latency machine control. Kyber uses a single QUIC (UDP-based) socket to multiplex video, audio, and control inputs (mouse, keyboard, gamepad) while maintaining clock synchronization across multiple sensors. This synchronization is critical for robotics AI training, where data coherence across multiple cameras and sensors must be preserved. The guest explains that existing solutions struggle when scaling beyond one camera.
Kyber uses forward error correction — over-transmitting a small percentage of data — to reconstruct lost packets without the latency penalty of TCP retransmission. A demo at CES showed a 3D-printed rover being controlled from France via a small PCB server. The current achieved latency is 7 milliseconds (Windows-to-Windows), with the encoder consuming ~3.5ms and decoder ~2ms. The ultimate goal is 4 milliseconds glass-to-glass over the internet. Kyber is dual-licensed: AGPL for open-source projects and a commercial license for proprietary products.
Key Insights
- The guest argues that adaptive streaming is fundamentally a CDN congestion problem rather than a video problem, and that the core buffering algorithm is quite basic — if a segment takes more than 50% of its download window to arrive, the player steps down in quality.
- The guest claims that audio quality degradation is more perceptually jarring than video quality drops, specifically noting that switching between full AAC and spectral band replication compressed AAC is immediately noticeable to listeners, while video resolution changes are smoother and less noticed.
- The guest explains that Kyber accounts for clock drift across multiple sensors on a robot, which existing solutions fail to handle beyond a single camera — a critical requirement for training robotics AI models on coherent, time-synchronized multi-sensor data.
- The guest states that Kyber uses forward error correction over a single QUIC (UDP-based) socket, intentionally over-transmitting a few percent of data so that lost packets can be reconstructed client-side without the latency cost of TCP-style retransmission acknowledgment.
- The guest reports that Kyber currently achieves 7 milliseconds glass-to-glass latency (Windows-to-Windows), with approximately 3.5ms consumed by the Nvidia hardware encoder and 2ms by the Intel decoder, leaving very little headroom to reach the 4ms goal without faster codecs or hardware.
Topics
Transcript
[0:02] I would love to sort of uh zoom in and and and talk a little bit more about the distinction between kind of downloading a file and watching it offline versus streaming. So the the complexities, the challenges of streaming. Is there something we could say about what it takes to uh stream files? We've been talking about codecs and I think a lot of that implies encoding and decoding uh without the having to communicate [0:33] over the network. >> Sure. >> Sure. >> Uh so can you can you elaborate like what's required to do over the network stuff? >> Yeah, but it is less complex than it seems compared to everything that we've talked about. Um…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
Do antipsychotic drugs work? | Andrew Scull and Lex Fridman
Andrew Scull discusses the efficacy and serious side effects of antipsychotic drugs, explaining that while they help some patients, many are non-responders, and the medications carry substantial risks including tardive dyskinesia, weight gain, metabolic syndrome, and cognitive dulling. The landmark CATIE study revealed that newer, expensive second-generation antipsychotics are no more effective than older first-generation drugs, with 67-82% of patients dropping out due to inefficacy or intolerable side effects.
Surprising origin of antipsychotic drugs | Andrew Scull and Lex Fridman
Andrew Scull explains how chlorpromazine, an antihistamine synthesized in the 1880s, was accidentally discovered to have psychiatric applications in the 1950s through French Navy lieutenant Alain Laborit's experiments. The drug's rapid adoption was driven by pharmaceutical companies' aggressive marketing to politicians and hospital administrators rather than by psychiatrists' initial enthusiasm, transforming it from a 'major tranquilizer' into the first 'antipsychotic' drug.
Cognitive Behavioral Therapy (CBT) vs Psychoanalysis | Andrew Scull and Lex Fridman
Andrew Scull discusses how clinical psychology and cognitive behavioral therapy (CBT) emerged after WWII as alternatives to psychoanalysis, becoming dominant through better empirical evidence, shorter treatment duration, and federal funding advantages. CBT's symptom-focused approach proved more measurable and reproducible than psychoanalytic theory, though its effectiveness remains limited for serious mental disorders.
Do anti-depressant drugs work? | Andrew Scull and Lex Fridman
Andrew Scull discusses the history and efficacy of antidepressants, particularly SSRIs like Prozac, explaining that while they statistically outperform placebo, the clinical improvement is often marginal and comes with significant side effects including sexual dysfunction and withdrawal difficulties. He also addresses 'diagnostic creep' in psychiatry, where conditions become increasingly broadly defined and diagnosed over time.
How drugs and psycho-pharmacology changed mental health | Andrew Scull and Lex Fridman
Andrew Scull traces the accidental discovery of psychopharmacology, beginning with chlorpromazine's unexpected psychiatric effects in the 1950s, and discusses how pharmaceutical companies transformed mental health treatment while also introducing serious side effects like tardive dyskinesia and metabolic complications that remain inadequately addressed.