RH20T

RH20T (Real-world Human-robot Teleoperation 20 Task) is a comprehensive robotic manipulation dataset developed by Shanghai Jiao Tong University (Hao-Shu Fang, Hongjie Fang, Cewu Lu, et al.), published in 2023. It contains over 110,000 contact-rich robot manipulation sequences collected across diverse skills, contexts, robot platforms, and camera viewpoints. The dataset was designed to enable one-shot imitation learning for complex, real-world manipulation skills. Unlike earlier datasets focused on simple push or pick-place tasks, RH20T targets multi-modal perception scenarios that require both visual and tactile feedback, reflecting the complexity of real-world manipulation. Data modalities include 1280x720 RGB and depth images at 10Hz, binocular IR images, robot joint angles and torques, gripper Cartesian pose at 100Hz, 6-DoF force/torque sensing at 100Hz, fingertip tactile sensing at 200Hz, and synchronized audio. RH20T covers a wide range of manipulation tasks from precise assembly and insertion to deformable object handling and tool use. Its multi-modal, multi-robot design makes it a valuable resource for training policies that can generalize across different sensing modalities and robot embodiments. The dataset has become an important resource for one-shot imitation learning and multi-modal policy learning, bridging the gap between simple visual-only manipulation datasets and the multi-sensory requirements of real-world robotic applications.