Using VR Controllers for Robot Teleoperation VR controllers (Meta Quest, HTC Vive, Valve Index, etc.) have become a popular input device for robot teleoperation, and for good reason: they provide precise 6-DOF tracking, built-in buttons/triggers for gripper control, and an intuitive mapping between human hand motion and robot end-effector motion. This tutorial covers how to integrate VR controllers into a teleoperation pipeline. Why VR Controllers? Compared to a spacemouse or gamepad, VR controllers give you: Direct spatial mapping — moving your hand forward moves the end-effector forward, with no mode-switching between axes Sub-millimeter to centimeter tracking accuracy, depending on the headset's tracking system Built-in trigger and grip buttons, which map naturally to gripper open/close Stereo visual feedback if you pair it with a headset display fed by the robot's cameras, enabling immersive teleoperation The trade-off is added system complexity: you now depend on a VR runtime, and you need to handle coordinate frame conversions between VR space and robot space carefully. Step 1: Reading Controller Pose and Input Most VR SDKs (OpenXR, SteamVR, Oculus/Meta SDK) expose controller pose as a position + quaternion, plus button/trigger states. Here's a generic wrapper: class VRController: def __init__(self, sdk_client, hand="right"): self.sdk_client = sdk_client self.hand = hand def read(self) -> dict: pose = self.sdk_client.get_controller_pose(self.hand) return { "position": pose.position, # (x, y, z) in VR world frame "orientation": pose.quaternion, # (x, y, z, w) "trigger": self.sdk_client.get_trigger_value(self.hand), "grip": self.sdk_client.get_grip_value(self.hand), "buttons": self.sdk_client.get_buttons(self.hand), } Step 2: Coordinate Frame Alignment This is the step most tutorials skip and where most bugs live. The VR headset's world frame, the controller's local frame, and the robot's base frame are all different, and none of them naturally line up. You need a calibration transform. A simple and effective approach is a one-time origin alignment: ask the operator to hold the controller at a known reference pose (e.g., directly above the robot's end-effector), then compute the transform between VR space and robot space at that moment. import numpy as np from scipy.spatial.transform import Rotation as R def compute_calibration(vr_pose_at_ref, robot_pose_at_ref): """Both poses given as (position, quaternion) at the same physical moment.""" vr_pos, vr_quat = vr_pose_at_ref robot_pos, robot_quat = robot_pose_at_ref R_vr = R.from_quat(vr_quat) R_robot = R.from_quat(robot_quat) R_offset = R_robot * R_vr.inv() t_offset = np.array(robot_pos) - R_offset.apply(vr_pos) return R_offset, t_offset def vr_to_robot_frame(vr_pos, vr_quat, R_offset, t_offset): R_vr = R.from_quat(vr_quat) robot_pos = R_offset.apply(vr_pos) + t_offset robot_quat = (R_offset * R_vr).as_quat() return robot_pos, robot_quat Run this calibration at the start of every session — controller tracking origins can shift between headset power cycles. Step 3: Delta-Pose Teleoperation Rather than mapping VR pose directly to robot pose 1:1 (which can be jarring if the operator's arm reach doesn't match the robot's workspace), use relative/delta control: press and hold the grip button to "engage" control, and only the change in controller pose since engagement is applied to the robot. class VRDeltaTeleop: def __init__(self, controller, robot, position_scale=1.0): self.controller = controller self.robot = robot self.position_scale = position_scale self.engaged = False self.ref_vr_pos = None self.ref_robot_pose = None def step(self): state = self.controller.read() engage = state["grip"] > 0.5 if engage and not self.engaged: # Just started engaging: capture reference poses self.ref_vr_pos = np.array(state["position"]) self.ref_robot_pose = self.robot.get_state()["ee_pose"] self.engaged = True elif not engage: self.engaged = False return None # no motion when disengaged delta = (np.array(state["position"]) - self.ref_vr_pos) * self.position_scale target_pose = self.ref_robot_pose.copy() target_pose[:3] += delta gripper_cmd = state["trigger"] return target_pose, gripper_cmd This "clutch" mechanism (engage/disengage like picking up a mouse to reposition it) is exactly how experienced teleoperators handle limited workspace — and it noticeably reduces fatigue over long collection sessions. Step 4: Adding Haptic and Visual Feedback If your VR SDK exposes haptics, use them to signal contact events (e.g., gripper closing on an object, or approaching a force limit read from the robot's F/T sensor): def send_contact_feedback(sdk_client, hand, force_magnitude): intensity = min(force_magnitude / 20.0, 1.0) # normalize to [0, 1] sdk_client.trigger_haptic_pulse(hand, duration_ms=50, amplitude=intensity) For visual feedback, streaming the robot's wrist camera to the headset display (rather than a fixed monitor) significantly improves depth perception and precision for fine manipulation tasks. Step 5: Integrating with the Data Collection Pipeline VR teleoperation slots directly into the teleoperation and logging pipeline from the earlier tutorials — swap in VRDeltaTeleop as the input source: def vr_teleop_loop(vr_teleop, robot, camera, logger, hz=20): dt = 1.0 / hz while True: start = time.time() result = vr_teleop.step() if result is not None: target_pose, gripper_cmd = result robot.send_command(target_pose, gripper_cmd) obs = {"image": camera.get_frame(), "ee_pose": robot.get_state()["ee_pose"]} logger.log(obs, target_pose, time.time()) time.sleep(max(0, dt - (time.time() - start))) Practical Tips Filter controller jitter. Raw VR tracking has small-scale noise; a simple exponential moving average on position/orientation smooths it out without adding noticeable lag. Cap velocity, not just position. Clamp the per-step delta to a maximum magnitude to prevent sudden large jumps from tracking glitches. Test calibration drift. Long sessions (30+ minutes) can accumulate small drift; consider a quick re-calibration checkpoint between recording batches. Mind the latency budget. VR-to-robot pipelines add tracking + wireless (if applicable) + rendering latency on top of robot control latency — measure end-to-end latency and keep it under ~100ms for comfortable operation. What's Next VR teleoperation is especially powerful when paired with sensor recording and replay for debugging — covered in the next tutorial — since you can visually replay exactly what the operator saw and did. Useful Links Website: www.v-modal.com SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter SDK Android: https://github.com/v-modal/vmodal_sdk_android Discord: https://discord.gg/K72z28KUx