Alignment-Enhanced Interactive Fusion Model for Complete and Incomplete Multimodal Hand Gesture Recognition.

Area of Science:

• Biomedical Engineering
• Human-Computer Interaction
• Machine Learning

Background:

• Hand gesture recognition (HGR) using surface electromyogram (sEMG) and accelerometer (ACC) signals is vital for advanced interfaces.
• Current neural network fusion methods often overlook hierarchical cross-modal information, leading to inefficient fusion.

Purpose of the Study:

• To propose a novel Alignment-Enhanced Interactive Fusion (AiFusion) model for effective multimodal HGR.
• To enable flexible and robust HGR for both complete and incomplete multimodal scenarios.

Main Methods:

• Developed a progressive hierarchical fusion strategy within the AiFusion model.
• Integrated two unimodal branches with a cascaded transformer-based multimodal fusion branch.
• Employed cross-modal supervised contrastive learning and online distillation for alignment.

Main Results:

• AiFusion demonstrated superior performance in complete multimodal HGR against state-of-the-art benchmarks.
• The model significantly outperformed unimodal baselines in challenging incomplete multimodal HGR.
• Experimental validation across five public datasets confirmed the model's effectiveness.

Conclusions:

• AiFusion offers an effective and robust solution for multimodal HGR.
• The proposed fusion strategy successfully integrates hierarchical cross-modal information.
• The model's ability to handle incomplete multimodal data addresses real-world challenges.