Fast robot teleoperation (at ultra-low 8ms latency): Kyber explained | Lex Fridman Podcast
Jean-Baptiste Kempf discusses Kyber, an open-source SDK platform for ultra-low latency remote machine control targeting 4ms glass-to-glass latency. The technology uses QUIC/UDP-based single-socket streaming with synchronized multi-stream support for robots, drones, and remote vehicles. Kyber is dual-licensed under AGPL and commercial licenses, drawing on lessons learned from VLC and broadcast engineering.
Summary
Jean-Baptiste Kempf, drawing on his background building VLC and serving as CTO of a cloud gaming startup, introduces Kyber — an open-source SDK platform designed to achieve ultra-low latency remote control of machines including robots, drones, submarines, and vehicles. The core motivation is that traditional video streaming optimizes for quality, but teleoperation requires optimizing for latency, where every millisecond matters for feedback-based control.
Kyber's technical architecture centers on a single QUIC/UDP socket that simultaneously carries video, audio, and control inputs (mouse, keyboard, gamepad) while maintaining synchronization across all streams. This avoids the head-of-line blocking problems inherent in TCP and HTTP/2. A key engineering challenge addressed is clock drift: when a robot has multiple cameras and sensors, ensuring all data is temporally coherent is critical both for real-time control and for collecting training data for AI models. Kyber implements server-side real-time stamping with clock drift compensation to guarantee data coherence during both operation and playback.
For reliability over unpredictable internet connections, Kyber uses forward error correction (FEC), over-transmitting a small percentage of data so that lost packets can be reconstructed client-side without adding the latency that acknowledgment-based retransmission would require. The team demonstrated the technology at CES by remotely controlling a 3D-printed rover with a telescopic arm from France, with the operator located on the other side of the planet.
On latency performance, the current achieved benchmark is 7ms glass-to-glass (Windows-to-Windows or Windows-to-Mac), broken down as approximately 3.5ms in the Nvidia hardware encoder and 2ms in the Intel decoder. The aspirational target is 4ms glass-to-glass, which corresponds to 240Hz refresh capability. Reaching that goal will likely require new codec approaches or faster hardware encoders.
Kempf envisions a future where millions of robots and drones are either teleoperated or 'teleobserved' — monitored by AI models that flag anomalies and escalate to human operators only when needed. Use cases span industrial inspection, defense, remote surgery, autonomous vehicles, and humanoid caregiving robots. Kyber is open source under a dual AGPL/commercial license: hobbyists and open-source projects can use it freely, while commercial entities that want to keep their products closed-source pay for a commercial license.
Key Insights
- Kempf argues that clock drift is an underappreciated problem in robotics: once a robot has five or six cameras, ensuring all sensor data is temporally synchronized becomes critical not just for real-time control but for AI training data coherence during replay.
- Kempf claims Kyber currently achieves 7ms glass-to-glass latency, with the encoder and decoder alone consuming ~5.5ms (3.5ms Nvidia hardware encoder + 2ms Intel decoder), meaning networking overhead is nearly zero — and the bottleneck to reaching the 4ms goal is the codec hardware, not the network.
- Kempf argues that forward error correction — over-transmitting a few percent of data so lost packets can be reconstructed client-side — is the correct tradeoff for teleoperation over the internet, because checking for full packet delivery adds too much latency.
- Kempf envisions that fully autonomous robots and vehicles will still require 'teleobservation' infrastructure: AI models continuously monitoring feeds and escalating to human operators only when anomalies are detected, making low-latency streaming a safety-critical requirement even in autonomous systems.
- Kempf describes Kyber's dual AGPL/commercial license as a deliberate strategy: hobbyists and open-source builders get free access, while large commercial companies that want closed-source usage pay for a commercial license — a model he explicitly learned from his experience with VLC licensing.
Topics
Transcript
[0:03] Speaking of networking and latency, so your new effort as we mentioned is Kyber, which is uh aimed at ultra-low latency. As you say, every millisecond counts, and uh you're applying that to remote control machines like robots, drones, computers. Can you tell me about it? Sure. Um if you start from where we used to be, right? You used to use FFmpeg to encode files, right? And then we used FFmpeg and VLC to encode in [0:34] streaming services, right? And then you need to go lower and lower. And the question was where to where we can can we go? Mhm. Um and this question is a very important because there are many use cases where you…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
Anti-matter & nuclear weapons: Why technology is always a double-edge sword | Don Lincoln
Don Lincoln discusses how advanced energy sources like nuclear fusion, fission, and antimatter represent transformative but double-edged technologies. He argues that science's role is to understand nature, while society must collectively decide how to apply that knowledge. The conversation concludes with a celebration of humanity's innate curiosity as the driver of civilizational progress.
Why antimatter costs $63 trillion dollars to produce | Don Lincoln and Lex Fridman
Don Lincoln and Lex Fridman discuss the extreme difficulty and cost of producing antimatter, noting that Fermilab could only produce about one nanogram per year. They explore the theoretical potential of antimatter as a propulsion system for space travel, while emphasizing that the core challenge is an engineering problem of concentrating energy, not a physics breakthrough.
Is loop quantum gravity wrong? - physicist explains | Don Lincoln and Lex Fridman
Don Lincoln explains the differences between loop quantum gravity and string theory, noting that loop quantum gravity attempts to quantize space itself rather than unify all forces. He discusses how an early prediction of loop quantum gravity — that light speed would vary by frequency — was disproven by gamma ray burst observations, but the theory adapted. He also highlights the landmark gravitational wave observation confirming that gravity travels at the speed of light.
Can antimatter be used as rocket fuel? | Don Lincoln and Lex Fridman
Don Lincoln and Lex Fridman discuss the feasibility of using antimatter as rocket fuel, noting that while it is physically possible, the cost of production (estimated at $62-63 trillion per gram) and containment challenges make it impractical. Lincoln explains that antimatter propulsion is fundamentally an engineering problem rather than a physics mystery, and that breakthroughs would likely come from finding new ways to concentrate energy rather than new physics theory.
Speed of light explained: Was Einstein's theory correct? | Don Lincoln and Lex Fridman
Don Lincoln explains Einstein's special relativity, focusing on the two core premises: the universality of natural laws and the constant speed of light for all observers. He describes modern particle physics experiments that have empirically confirmed Einstein's conjecture. He also reflects on how understanding space-time makes the concept of a universal speed limit intuitive rather than bizarre.