Evaluating dual-path temporal fusion strategies for multi-modal hand gesture recognition under limb-position...

Area of Science:

• Biomedical Engineering
• Human-Computer Interaction
• Machine Learning

Background:

• Wearable biosignal-based hand gesture recognition (HGR) is crucial for prosthetic hand control.
• Limb-position variability and real-world factors challenge HGR reliability.
• Co-located surface electromyography (sEMG) and pressure-based force myography (pFMG) offer complementary sensing.

Purpose of the Study:

• To systematically compare dual-path temporal fusion architectures for sEMG and pFMG integration.
• To evaluate architectures based on robustness, computational efficiency, and interpretability for prosthetic use.
• To identify optimal fusion strategies for wearable HMI systems.

Main Methods:

• Investigated three dual-path Temporal Convolutional Network (DFF-TCN) architectures: concatenation-based, decision-level cross-attention, and feature-level cross-attention.
• Evaluated models on a custom dataset from ten participants performing nine hand gestures across various arm positions.
• Utilized Integrated Gradients for explainable AI analysis of sEMG and pFMG contributions.

Main Results:

• The concatenation-based DFF-TCN achieved the highest mean accuracy (95.88%) and lowest inference latency (1.70 ms).
• Attention-based variants showed slightly lower accuracies (90.65% and 94.02%).
• Explainable AI revealed complementary contributions from sEMG (54.08%) and pFMG (45.92%).

Conclusions:

• Different fusion strategies present trade-offs between recognition performance, computational cost, and robustness.
• The concatenation-based model offers a practical balance for real-time prosthetic control.
• Findings guide the selection of multi-modal fusion architectures for wearable HMI and prosthetic applications.