Fast robot teleoperation (at ultra-low 8ms latency): Kyber explained | Lex Fridman Podcast
Jean-Baptiste Kempf discusses Kyber, an open-source SDK platform for ultra-low latency remote machine control targeting 4ms glass-to-glass latency. The technology uses QUIC/UDP-based single-socket streaming with synchronized multi-stream support for robots, drones, and remote vehicles. Kyber is dual-licensed under AGPL and commercial licenses, drawing on lessons learned from VLC and broadcast engineering.
Summary
Jean-Baptiste Kempf, drawing on his background building VLC and serving as CTO of a cloud gaming startup, introduces Kyber — an open-source SDK platform designed to achieve ultra-low latency remote control of machines including robots, drones, submarines, and vehicles. The core motivation is that traditional video streaming optimizes for quality, but teleoperation requires optimizing for latency, where every millisecond matters for feedback-based control.
Kyber's technical architecture centers on a single QUIC/UDP socket that simultaneously carries video, audio, and control inputs (mouse, keyboard, gamepad) while maintaining synchronization across all streams. This avoids the head-of-line blocking problems inherent in TCP and HTTP/2. A key engineering challenge addressed is clock drift: when a robot has multiple cameras and sensors, ensuring all data is temporally coherent is critical both for real-time control and for collecting training data for AI models. Kyber implements server-side real-time stamping with clock drift compensation to guarantee data coherence during both operation and playback.
For reliability over unpredictable internet connections, Kyber uses forward error correction (FEC), over-transmitting a small percentage of data so that lost packets can be reconstructed client-side without adding the latency that acknowledgment-based retransmission would require. The team demonstrated the technology at CES by remotely controlling a 3D-printed rover with a telescopic arm from France, with the operator located on the other side of the planet.
On latency performance, the current achieved benchmark is 7ms glass-to-glass (Windows-to-Windows or Windows-to-Mac), broken down as approximately 3.5ms in the Nvidia hardware encoder and 2ms in the Intel decoder. The aspirational target is 4ms glass-to-glass, which corresponds to 240Hz refresh capability. Reaching that goal will likely require new codec approaches or faster hardware encoders.
Kempf envisions a future where millions of robots and drones are either teleoperated or 'teleobserved' — monitored by AI models that flag anomalies and escalate to human operators only when needed. Use cases span industrial inspection, defense, remote surgery, autonomous vehicles, and humanoid caregiving robots. Kyber is open source under a dual AGPL/commercial license: hobbyists and open-source projects can use it freely, while commercial entities that want to keep their products closed-source pay for a commercial license.
Key Insights
- Kempf argues that clock drift is an underappreciated problem in robotics: once a robot has five or six cameras, ensuring all sensor data is temporally synchronized becomes critical not just for real-time control but for AI training data coherence during replay.
- Kempf claims Kyber currently achieves 7ms glass-to-glass latency, with the encoder and decoder alone consuming ~5.5ms (3.5ms Nvidia hardware encoder + 2ms Intel decoder), meaning networking overhead is nearly zero — and the bottleneck to reaching the 4ms goal is the codec hardware, not the network.
- Kempf argues that forward error correction — over-transmitting a few percent of data so lost packets can be reconstructed client-side — is the correct tradeoff for teleoperation over the internet, because checking for full packet delivery adds too much latency.
- Kempf envisions that fully autonomous robots and vehicles will still require 'teleobservation' infrastructure: AI models continuously monitoring feeds and escalating to human operators only when anomalies are detected, making low-latency streaming a safety-critical requirement even in autonomous systems.
- Kempf describes Kyber's dual AGPL/commercial license as a deliberate strategy: hobbyists and open-source builders get free access, while large commercial companies that want closed-source usage pay for a commercial license — a model he explicitly learned from his experience with VLC licensing.
Topics
Transcript
[0:03] Speaking of networking and latency, so your new effort as we mentioned is Kyber, which is uh aimed at ultra-low latency. As you say, every millisecond counts, and uh you're applying that to remote control machines like robots, drones, computers. Can you tell me about it? Sure. Um if you start from where we used to be, right? You used to use FFmpeg to encode files, right? And then we used FFmpeg and VLC to encode in [0:34] streaming services, right? And then you need to go lower and lower. And the question was where to where we can can we go? Mhm. Um and this question is a very important because there are many use cases where you…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
How the Civil War ended | Gary Gallagher and Lex Fridman
Gary Gallagher and Lex Fridman discuss how the Civil War ended, focusing on Lee's surrender at Appomattox and Grant's magnanimous treatment of Confederate forces. They explore how Grant's generous surrender terms, influenced by Lincoln's reconciliation goals, contrasted with calls from Radical Republicans for harsher punishment and how this approach facilitated national reunification rather than perpetuating resentment.
White supremacy after the Civil War | Gary Gallagher and Lex Fridman
Historian Gary Gallagher discusses Reconstruction, white supremacy, and Confederate memory in post-Civil War America, emphasizing that Northern troops were insufficient to enforce equal rights for freed Black people, and explaining how the "Lost Cause" narrative allowed the South to reframe the war's purpose while Jim Crow became the mechanism for maintaining white supremacy.
The truth about the Civil War - Why people volunteered to fight | Gary Gallagher and Lex Fridman
Historian Gary Gallagher discusses the distinction between causation and motivation in Civil War volunteerism, explaining that while slavery caused the war, most Northern soldiers were motivated by belief in preserving the Union as a democratic republic. He emphasizes that understanding citizen-soldiers' sense of civic duty and American exceptionalism is essential to comprehending why ordinary people risked their lives for the Union.
Christianity and religion during the Civil War | Gary Gallagher and Lex Fridman
Both Union and Confederate sides during the American Civil War were overwhelmingly Protestant Christian and used religious justification for their military efforts. Interestingly, Confederate leaders like Lee and Jackson were notably more overtly religious in their official communications than Union commanders like Grant and Sherman.
Racism in America after slavery was abolished | Gary Gallagher and Lex Fridman
Gary Gallagher discusses how abolishing slavery was only the beginning of achieving equal rights for Black Americans, noting that most Northern whites prioritized saving the Union over ensuring racial equality. He explains that legal changes like the Civil Rights Act took over a century to achieve, while the deeper work of changing societal attitudes and prejudices remains ongoing and vulnerable to exploitation.