Efficient Predictive VR Interaction for Fine Teleoperation Using Physics-Based simulation and automatic planning

In complex and hazardous environments, robots often require human's fine teleoperation to ensure accurate decision-making and optimal manipulation. To improve teleoperation efficiency, we design and develop 3D predictive Virtual Reality (VR) interfaces with four distinct automation levels. These 3D interfaces, coupled with physics-based simulation and task-level automatic planning, assist users in controlling robots for fine manipulation. Our comparative user study evaluates the efficiency of these interaction strategies, investigating the feasibility of solely relying on simulation results for robot control. We further compare VR controller-based teleoperation with direct hand gesture manipulation, assessing efficiency and user interaction experience. Our results demonstrate that predictive interfaces and intelligent assistance significantly improve operational efficiency. VR controller-based teleoperation outperforms hand gesture interaction in terms of efficiency and user experience for fine manipulation. In particular, even novice users can effectively utilize the predictive VR interface for precise remote robot control. This work presents a novel approach of integrating predictive interaction, physics-based simulation, and task-level automatic planning for fine teleoperation.