Mixed Reality Robotic Teleoperation
A Quest 3 app that places a Franka arm on your real desk in passthrough, so you can drive it by hand. Isaac Lab owns all the physics.
- Role
- Simulation & XR Engineer
- Company
- Independent R&D
- Period
- 2026 – ongoing
- Status
- In progress
- 72 fpsPassthrough teleop on Quest 3
- ≤ 0.01 mmIsaac ⇄ Unity agreement
- 0 / 4591States dropped
What I did
- Native Unity 6 mixed-reality app on Quest 3. Place the robot on your desk with a controller or an open hand. The arm, table and task cubes render in passthrough at 72 fps.
- Isaac Lab is the only physics authority. Unity just applies a 20 Hz state stream and sends hand/controller poses back at 72 Hz over one WebSocket.
- Verified frame transform: every state is re-checked against the simulator. Measured ≤ 0.01 mm / 0.00°, with 0 of 4,591 states dropped.
- Hand tracking as a first-class input: palm-driven control, a One Euro filter (±3 mm → ~1.4 mm jitter), and a gesture clutch to cover the full reach.
- Jitter-free playback on the simulator's step clock, over USB or Wi-Fi (~20 ms round trip).
- Data pipeline: episodes are recorded to robomimic HDF5 and multiplied with Isaac Lab Mimic to train Behaviour Cloning and Diffusion Policy. A spin-off CLI, Sift, scores LeRobot v3.0 datasets per episode.