Fast robot teleoperation (at ultra-low 8ms latency): Kyber explained | Lex Fridman Podcast
Jean-Baptiste Kempf discusses Kyber, an open-source SDK platform for ultra-low latency remote machine control targeting 4ms glass-to-glass latency. The technology uses QUIC/UDP-based single-socket streaming with synchronized multi-stream support for robots, drones, and remote vehicles. Kyber is dual-licensed under AGPL and commercial licenses, drawing on lessons learned from VLC and broadcast engineering.
Summary
Jean-Baptiste Kempf, drawing on his background building VLC and serving as CTO of a cloud gaming startup, introduces Kyber — an open-source SDK platform designed to achieve ultra-low latency remote control of machines including robots, drones, submarines, and vehicles. The core motivation is that traditional video streaming optimizes for quality, but teleoperation requires optimizing for latency, where every millisecond matters for feedback-based control.
Kyber's technical architecture centers on a single QUIC/UDP socket that simultaneously carries video, audio, and control inputs (mouse, keyboard, gamepad) while maintaining synchronization across all streams. This avoids the head-of-line blocking problems inherent in TCP and HTTP/2. A key engineering challenge addressed is clock drift: when a robot has multiple cameras and sensors, ensuring all data is temporally coherent is critical both for real-time control and for collecting training data for AI models. Kyber implements server-side real-time stamping with clock drift compensation to guarantee data coherence during both operation and playback.
For reliability over unpredictable internet connections, Kyber uses forward error correction (FEC), over-transmitting a small percentage of data so that lost packets can be reconstructed client-side without adding the latency that acknowledgment-based retransmission would require. The team demonstrated the technology at CES by remotely controlling a 3D-printed rover with a telescopic arm from France, with the operator located on the other side of the planet.
On latency performance, the current achieved benchmark is 7ms glass-to-glass (Windows-to-Windows or Windows-to-Mac), broken down as approximately 3.5ms in the Nvidia hardware encoder and 2ms in the Intel decoder. The aspirational target is 4ms glass-to-glass, which corresponds to 240Hz refresh capability. Reaching that goal will likely require new codec approaches or faster hardware encoders.
Kempf envisions a future where millions of robots and drones are either teleoperated or 'teleobserved' — monitored by AI models that flag anomalies and escalate to human operators only when needed. Use cases span industrial inspection, defense, remote surgery, autonomous vehicles, and humanoid caregiving robots. Kyber is open source under a dual AGPL/commercial license: hobbyists and open-source projects can use it freely, while commercial entities that want to keep their products closed-source pay for a commercial license.
Key Insights
- Kempf argues that clock drift is an underappreciated problem in robotics: once a robot has five or six cameras, ensuring all sensor data is temporally synchronized becomes critical not just for real-time control but for AI training data coherence during replay.
- Kempf claims Kyber currently achieves 7ms glass-to-glass latency, with the encoder and decoder alone consuming ~5.5ms (3.5ms Nvidia hardware encoder + 2ms Intel decoder), meaning networking overhead is nearly zero — and the bottleneck to reaching the 4ms goal is the codec hardware, not the network.
- Kempf argues that forward error correction — over-transmitting a few percent of data so lost packets can be reconstructed client-side — is the correct tradeoff for teleoperation over the internet, because checking for full packet delivery adds too much latency.
- Kempf envisions that fully autonomous robots and vehicles will still require 'teleobservation' infrastructure: AI models continuously monitoring feeds and escalating to human operators only when anomalies are detected, making low-latency streaming a safety-critical requirement even in autonomous systems.
- Kempf describes Kyber's dual AGPL/commercial license as a deliberate strategy: hobbyists and open-source builders get free access, while large commercial companies that want closed-source usage pay for a commercial license — a model he explicitly learned from his experience with VLC licensing.
Topics
Full transcript available for MurmurCast members
Sign Up to Access