Research on Deep Learning-Based Human-Robot Static/Dynamic Gesture-Driven Control Framework

Area of Science:

• Robotics
• Artificial Intelligence
• Computer Vision

Background:

• Human-robot interaction (HRI) is crucial for collaborative tasks.
• Gesture-driven control offers a natural interface for robots.
• Robustness in varying conditions is essential for practical HRI.

Purpose of the Study:

• To develop a deep learning-based system for gesture-driven robot control.
• To enable robots to perform object-grasping and delivery tasks using static and dynamic hand gestures.
• To evaluate the system's performance and robustness in diverse lighting conditions.

Main Methods:

• Utilized 2D Convolutional Neural Networks (2D-CNNs) for static gesture recognition.
• Employed a hybrid 3D Convolutional Neural Networks (3D-CNNs) and Long Short-Term Memory (3D-CNN+LSTM) network for dynamic gesture recognition.
• Integrated MediaPipe for hand feature extraction and a depth camera for 3D pose estimation.

Main Results:

• Achieved validation accuracies of 95.38% for static and 93.18% for dynamic gestures.
• Demonstrated average task success rates of no less than 96.88% (static) and 94.63% (dynamic) across 100 trials per participant.
• Maintained task completion times consistently within 20 seconds under natural, low, and strong light conditions.

Conclusions:

• The proposed deep learning approach enables robust vision-based robotic control using natural hand gestures.
• The system effectively facilitates object-grasping and delivery tasks, showing high accuracy and reliability.
• This research holds significant promise for advancing human-robot collaboration applications.