A multimodal multi-label deep learning framework with temporal attention for detection of upper-limb motor activities...

Area of Science:

• Biomedical Engineering
• Machine Learning
• Rehabilitation Technology

Background:

• Accurate detection of simultaneous grip and elbow flexion is crucial for advanced rehabilitation, prosthetic control, and human-machine interaction.
• Existing methods, primarily using surface electromyography (sEMG), face challenges in capturing overlapping muscle activations and biomechanical context during dynamic upper-limb movements.

Purpose of the Study:

• To develop and evaluate a multimodal, multi-label deep learning framework for concurrent detection of grip and elbow flexion.
• To integrate sEMG, pressure distribution, and inertial sensing for enhanced movement detection.

Main Methods:

• A multimodal deep learning framework utilizing a convolutional neural network (CNN) and bidirectional long-short-term memory (BiLSTM) with temporal attention.
• Integration of data from three sEMG channels, four pressure sensors, and a nine-axis inertial measurement unit.
• Synchronized, standardized, and segmented data into fixed-length windows for processing.

Main Results:

• The proposed framework achieved a window-level macro F1-score of 0.9573 and a file-level macro F1-score of 0.9783.
• Feature analysis revealed dominant contributions from inertial and pressure signals, with sEMG providing complementary information.
• The system demonstrated robust multi-label detection capabilities for complex upper-limb movements.

Conclusions:

• The multimodal sensor fusion approach significantly improves the accuracy and robustness of detecting simultaneous upper-limb movements compared to single-modality systems.
• This framework addresses key limitations of traditional sEMG-based systems.
• The developed system supports scalable real-time applications in rehabilitation and assistive technologies.