Deep learning multimodal fNIRS and EEG signals for bimanual grip force decoding.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Rehabilitation Technology

Background:

• Non-invasive brain-machine interfaces (BMIs) enable environmental interaction but require force decoding for stability.
• Decoding unimanual forces is established, but bimanual force control is needed for broader BMI applications.
• Hand-specific force decoding is essential for advanced BMI control and reducing signal crosstalk.

Purpose of the Study:

• To investigate the decoding of hand-specific forces using combined electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS).
• To develop and evaluate a deep-learning architecture for improved fusion of multimodal neuroimaging data.
• To enhance the robustness and interpretability of BMI systems for bimanual force control.

Main Methods:

• Utilized a deep-learning architecture with attention and residual layers (cnnatt) for EEG and fNIRS signal fusion.
• Participants generated hand-specific force profiles to train and test decoding models.
• Compared the performance of the deep-learning model against linear decoders for bimanual force decoding.

Main Results:

• Combined EEG and fNIRS improved bimanual force decoding accuracy compared to individual modalities.
• Deep-learning models, particularly cnnatt, outperformed linear models in force decoding.
• Force generation detection was significantly improved and showed hand-specificity, especially for the dominant hand.
• The cnnatt model revealed distinct cortical encoding patterns for forces from each hand and force-level modulation of neural activity.

Conclusions:

• The developed deep-learning approach effectively fuses EEG and fNIRS for accurate hand-specific force decoding.
• Improved bimanual force control in BMIs can be achieved by addressing hand-specific neural signals.
• These findings advance BMI technology for more robust robotic control and offer new insights for motor rehabilitation assessment.