Fast robot teleoperation (at ultra-low 8ms latency): Kyber explained | Lex Fridman Podcast
Jean-Baptiste Kempf discusses Kyber, an open-source SDK platform for ultra-low latency remote machine control targeting 4ms glass-to-glass latency. The technology uses QUIC/UDP-based single-socket streaming with synchronized multi-stream support for robots, drones, and remote vehicles. Kyber is dual-licensed under AGPL and commercial licenses, drawing on lessons learned from VLC and broadcast engineering.
Summary
Jean-Baptiste Kempf, drawing on his background building VLC and serving as CTO of a cloud gaming startup, introduces Kyber — an open-source SDK platform designed to achieve ultra-low latency remote control of machines including robots, drones, submarines, and vehicles. The core motivation is that traditional video streaming optimizes for quality, but teleoperation requires optimizing for latency, where every millisecond matters for feedback-based control.
Kyber's technical architecture centers on a single QUIC/UDP socket that simultaneously carries video, audio, and control inputs (mouse, keyboard, gamepad) while maintaining synchronization across all streams. This avoids the head-of-line blocking problems inherent in TCP and HTTP/2. A key engineering challenge addressed is clock drift: when a robot has multiple cameras and sensors, ensuring all data is temporally coherent is critical both for real-time control and for collecting training data for AI models. Kyber implements server-side real-time stamping with clock drift compensation to guarantee data coherence during both operation and playback.
For reliability over unpredictable internet connections, Kyber uses forward error correction (FEC), over-transmitting a small percentage of data so that lost packets can be reconstructed client-side without adding the latency that acknowledgment-based retransmission would require. The team demonstrated the technology at CES by remotely controlling a 3D-printed rover with a telescopic arm from France, with the operator located on the other side of the planet.
On latency performance, the current achieved benchmark is 7ms glass-to-glass (Windows-to-Windows or Windows-to-Mac), broken down as approximately 3.5ms in the Nvidia hardware encoder and 2ms in the Intel decoder. The aspirational target is 4ms glass-to-glass, which corresponds to 240Hz refresh capability. Reaching that goal will likely require new codec approaches or faster hardware encoders.
Kempf envisions a future where millions of robots and drones are either teleoperated or 'teleobserved' — monitored by AI models that flag anomalies and escalate to human operators only when needed. Use cases span industrial inspection, defense, remote surgery, autonomous vehicles, and humanoid caregiving robots. Kyber is open source under a dual AGPL/commercial license: hobbyists and open-source projects can use it freely, while commercial entities that want to keep their products closed-source pay for a commercial license.
Key Insights
- Kempf argues that clock drift is an underappreciated problem in robotics: once a robot has five or six cameras, ensuring all sensor data is temporally synchronized becomes critical not just for real-time control but for AI training data coherence during replay.
- Kempf claims Kyber currently achieves 7ms glass-to-glass latency, with the encoder and decoder alone consuming ~5.5ms (3.5ms Nvidia hardware encoder + 2ms Intel decoder), meaning networking overhead is nearly zero — and the bottleneck to reaching the 4ms goal is the codec hardware, not the network.
- Kempf argues that forward error correction — over-transmitting a few percent of data so lost packets can be reconstructed client-side — is the correct tradeoff for teleoperation over the internet, because checking for full packet delivery adds too much latency.
- Kempf envisions that fully autonomous robots and vehicles will still require 'teleobservation' infrastructure: AI models continuously monitoring feeds and escalating to human operators only when anomalies are detected, making low-latency streaming a safety-critical requirement even in autonomous systems.
- Kempf describes Kyber's dual AGPL/commercial license as a deliberate strategy: hobbyists and open-source builders get free access, while large commercial companies that want closed-source usage pay for a commercial license — a model he explicitly learned from his experience with VLC licensing.
Topics
Transcript
[0:03] Speaking of networking and latency, so your new effort as we mentioned is Kyber, which is uh aimed at ultra-low latency. As you say, every millisecond counts, and uh you're applying that to remote control machines like robots, drones, computers. Can you tell me about it? Sure. Um if you start from where we used to be, right? You used to use FFmpeg to encode files, right? And then we used FFmpeg and VLC to encode in [0:34] streaming services, right? And then you need to go lower and lower. And the question was where to where we can can we go? Mhm. Um and this question is a very important because there are many use cases where you…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Lex Clips
Do antipsychotic drugs work? | Andrew Scull and Lex Fridman
Andrew Scull discusses the efficacy and serious side effects of antipsychotic drugs, explaining that while they help some patients, many are non-responders, and the medications carry substantial risks including tardive dyskinesia, weight gain, metabolic syndrome, and cognitive dulling. The landmark CATIE study revealed that newer, expensive second-generation antipsychotics are no more effective than older first-generation drugs, with 67-82% of patients dropping out due to inefficacy or intolerable side effects.
Surprising origin of antipsychotic drugs | Andrew Scull and Lex Fridman
Andrew Scull explains how chlorpromazine, an antihistamine synthesized in the 1880s, was accidentally discovered to have psychiatric applications in the 1950s through French Navy lieutenant Alain Laborit's experiments. The drug's rapid adoption was driven by pharmaceutical companies' aggressive marketing to politicians and hospital administrators rather than by psychiatrists' initial enthusiasm, transforming it from a 'major tranquilizer' into the first 'antipsychotic' drug.
Cognitive Behavioral Therapy (CBT) vs Psychoanalysis | Andrew Scull and Lex Fridman
Andrew Scull discusses how clinical psychology and cognitive behavioral therapy (CBT) emerged after WWII as alternatives to psychoanalysis, becoming dominant through better empirical evidence, shorter treatment duration, and federal funding advantages. CBT's symptom-focused approach proved more measurable and reproducible than psychoanalytic theory, though its effectiveness remains limited for serious mental disorders.
Do anti-depressant drugs work? | Andrew Scull and Lex Fridman
Andrew Scull discusses the history and efficacy of antidepressants, particularly SSRIs like Prozac, explaining that while they statistically outperform placebo, the clinical improvement is often marginal and comes with significant side effects including sexual dysfunction and withdrawal difficulties. He also addresses 'diagnostic creep' in psychiatry, where conditions become increasingly broadly defined and diagnosed over time.
How drugs and psycho-pharmacology changed mental health | Andrew Scull and Lex Fridman
Andrew Scull traces the accidental discovery of psychopharmacology, beginning with chlorpromazine's unexpected psychiatric effects in the 1950s, and discusses how pharmaceutical companies transformed mental health treatment while also introducing serious side effects like tardive dyskinesia and metabolic complications that remain inadequately addressed.