Teleoperation System for Service Robots Using a Virtual Reality Headset and 3D Pose Estimation

Area of Science:

• Robotics
• Human-Computer Interaction
• Computer Vision

Background:

• Service robots require intuitive control for complex tasks.
• Existing teleoperation methods often lack natural interaction and situational awareness.
• Integrating real-time human pose estimation with VR offers a promising solution.

Purpose of the Study:

• To develop an immersive teleoperation framework for service robots.
• To enable intuitive and natural robot control using 3D human pose estimation and VR.
• To enhance situational awareness and robot manipulation capabilities.

Main Methods:

• Utilized MediaPipe for 2D landmark detection and an Intel RealSense D455 camera for depth data.
• Reconstructed 3D human joint positions and computed joint angles.
• Mapped joint angles to the CHARMIE service robot's anthropomorphic arm kinematics.
• Implemented a VR interface with stereoscopic video and head-motion control.

Main Results:

• Demonstrated reliable arm teleoperation and effective telepresence in real-world object grasping tasks.
• Achieved intuitive control of the service robot's arm.
• Identified limitations in vision-only estimation for axial joint rotations, especially with occlusions.

Conclusions:

• The proposed framework offers a practical approach to low-cost, sensor-driven, immersive human-robot interaction.
• The system enhances service robot capabilities in dynamic environments.
• Further research is needed to address limitations in precise joint rotation estimation for improved teleoperation.