An adaptive reinforcement learning-based multimodal data fusion framework for human-robot confrontation gaming.

Area of Science:

• Robotics
• Artificial Intelligence
• Human-Robot Interaction

Background:

• Human-robot confrontation (HRC) applications, such as game playing, are increasingly common.
• Existing methods struggle with robot intelligence and motion capture system interference.
• There is a need for robust frameworks to improve HRC performance.

Purpose of the Study:

• To present an adaptive reinforcement learning (RL) based multimodal data fusion (AdaRL-MDF) framework.
• To enhance robot intelligence and anti-interference capabilities in HRC.
• To enable a robot hand to play Rock-Paper-Scissors (RPS) with humans.

Main Methods:

• Developed an AdaRL-MDF framework integrating an adaptive learning mechanism, an RL model, and multimodal data fusion.
• Employed an ensemble classifier combining k-nearest neighbor (k-NN) and deep convolutional neural network (DCNN).
• Utilized depth vision for gesture recognition with a k-NN classifier.

Main Results:

• The AdaRL-MDF model demonstrated enhanced robot intelligence and interference resistance.
• The ensemble k-NN and DCNN model showed high performance in accuracy and computational time.
• The depth vision-based k-NN classifier achieved 100% identification accuracy for gestures.

Conclusions:

• The AdaRL-MDF framework significantly improves HRC applications by enhancing robot intelligence and robustness.
• The combined k-NN and DCNN ensemble model offers superior performance for gesture recognition.
• This research paves the way for more sophisticated and intelligent HRC systems.