A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision

Area of Science:

• Robotics
• Computer Vision
• Human-Robot Interaction

Background:

• Markerless vision-based teleoperation allows natural finger motions for robot hands.
• Current pose estimation methods struggle with finger self-occlusion, limiting accuracy.

Purpose of the Study:

• To develop a novel vision-based hand-arm teleoperation system to improve pose estimation accuracy.
• To enable natural and noninvasive control of multifingered robot hands.

Main Methods:

• Developed an end-to-end hand pose regression network (Transteleop) using image-to-image translation.
• Integrated an auxiliary reconstruction loss function for enhanced accuracy.
• Implemented a controlled active vision system using a robot arm for optimal hand observation.

Main Results:

• The system accurately captures human hand poses and predicts robot joint commands.
• The active vision system ensures high accuracy for the neural network.
• Demonstrated practicality and stability through complex manipulation tasks like tower building and cup stacking.

Conclusions:

• The proposed vision-based teleoperation system effectively overcomes self-occlusion challenges.
• It enables precise and natural control of robot hands for various tasks.
• The system offers a practical and stable solution for advanced robotic manipulation.