Eye-Movement-Assisted Time-Frequency EEG Decoding for Multimodal Robotic Arm Control

Area of Science:

• Neuroscience
• Human-Computer Interaction
• Biomedical Engineering

Background:

• Single-modality electroencephalography-based motor imagery (EEG-MI) faces interference and limited accuracy, hindering practical brain-computer interface (BCI) applications.
• Existing algorithmic models struggle with classification accuracy and actionable control commands for interactive devices.

Purpose of the Study:

• To develop a multimodal human-computer interaction control scheme integrating eye-movement command encoding with EEG motor imagery decoding.
• To enhance the EEG-TransNet model for improved multi-domain feature representation and fusion.
• To establish self-collected EEG-MI and eye-movement datasets for validating the multimodal control framework.

Main Methods:

• Integrated eye movements as discrete commands (start, stop, grasp, release) to reduce EEG-MI decoding burden.
• Enhanced EEG-TransNet with a time-frequency feature branch and an adaptive multi-branch EEG feature gating module.
• Utilized four independent SVM binary classifiers for eye movement pattern identification and binary-encoded EEG/eye-movement results for hardware control.

Main Results:

• Achieved average classification accuracies of 86.96% (BCI IV-2a) and 88.73% (self-collected EEG).
• Demonstrated an average task completion time of 17 seconds in robotic-arm grasping experiments.
• Provided preliminary evidence for the real-time feasibility of the multimodal control framework.

Conclusions:

• The proposed multimodal BCI control scheme effectively enhances control accuracy and command generation for assistive technologies.
• The enhanced EEG-TransNet model shows improved performance in processing complex EEG signals.
• The integration of eye-movement commands offers a promising direction for practical and robust BCI applications.