A multimodal multi-label deep learning framework with temporal attention for detection of upper-limb motor activities...

Accurate detection of simultaneous grip and elbow flexion is critical for advanced rehabilitation monitoring, intuitive prosthetic control and human-machine interaction. Existing approaches, particularly those relying solely on surface electromyography (sEMG), often struggle to capture overlapping muscle activations and the biomechanical context of dynamic upper-limb movements. This study proposes a multimodal, multi-label deep learning framework integrating sEMG, pressure distribution and inertial sensing for concurrent detection of grip and elbow flexion. Signals were acquired from three sEMG channels, four pressure sensors embedded in a Wii Balance Board and a nine-axis inertial measurement unit. All modalities were synchronized, standardized and segmented into fixed-length windows of 100 samples. A customized convolutional neural network and bidirectional long-short-term memory architecture with temporal attention was employed to model spatial-temporal dependencies across sensor streams. Multi-label sigmoid outputs enabled simultaneous prediction of both movements. Evaluation was conducted using subject-level cross-validation with file-level train-test splitting. The proposed framework achieved a window-level macro F1-score of 0.9573 with a Hamming loss of 0.0674 and file-level aggregation yielded a macro F1-score of 0.9783. Feature analysis indicated dominant contributions from inertial and pressure signals with sEMG providing complementary activation information. The proposed multimodal sensor fusion enables robust multi-label detection with attention-guided temporal weighting of complex upper-limb movements. This framework helps address key limitations of single-modality EMG systems and supports scalable real-time rehabilitation and assistive applications.