Telepresence augmentation for visual and haptic guided immersive teleoperation of industrial manipulator

Teleoperation under human guidance has become an effective solution to extend human's reach in various environments. However, the teleoperation system still faces challenges of insufficient sense of both visual and haptic feedback from remote environments, which results in the inadequate guidance for the operator. In this paper, a visual/haptic integrated perception and reconstruction system (VHI-PRS) is developed to provide the operator with 3D visual information and effective haptic guidance. Specifically, a visual-based telepresence augmentation method is proposed to provide the operator with virtual-reality combined visual telepresence, where the real point cloud model is directly superimposed on virtual manipulator to avoid the time-consuming process of mesh model rendering. With the utilization of visual information, a haptic-based telepresence augmentation method is proposed to provide the operator with comprehensive force feedback, including the virtual guiding force, virtual repulsive force and real-time interactive force, which greatly helps reduce the workload of operator. Finally, a user study on grab-place task is carried out to verify the effectiveness of proposed system.