Cross-modal guiding and reweighting network for multi-modal RSVP-based target detection

Area of Science:

• Neuroscience
• Computer Science
• Biomedical Engineering

Background:

• Rapid Serial Visual Presentation (RSVP) Brain-Computer Interfaces (BCI) detect rare target images using event-related potentials (ERPs).
• Current RSVP-BCI systems face limitations in decoding accuracy, hindering practical application.
• Integrating additional data modalities is crucial for enhancing RSVP-BCI performance.

Purpose of the Study:

• To introduce eye movement (EYE) data as a complementary modality to EEG for RSVP target detection.
• To develop and evaluate a novel multi-modal BCI system combining EEG and EYE data.
• To improve the decoding accuracy and practical utility of RSVP-based target detection systems.

Main Methods:

• Simultaneous recording of EEG signals and eye movements (gaze, pupil) during an RSVP target detection task.
• Construction of a multi-modal dataset comprising 20 subjects.
• Proposal of a cross-modal guiding and fusion network incorporating a two-branch backbone, Cross-Modal Feature Guiding (CMFG), Multi-scale Multi-modal Reweighting (MMR), and Dual Activation Fusion (DAF) modules.

Main Results:

• The proposed multi-modal network achieved a balanced accuracy of 88.00% (±2.29) on the RSVP dataset.
• Ablation studies and visualizations confirmed the effectiveness of the CMFG, MMR, and DAF modules.
• The EYE modality was demonstrated to be a valuable addition to EEG for RSVP tasks.

Conclusions:

• Introducing eye movement data significantly enhances RSVP-based BCI performance.
• The proposed cross-modal guiding and fusion network effectively integrates EEG and EYE modalities for superior RSVP decoding.
• This multi-modal approach offers a promising advancement for RSVP target detection systems, improving accuracy and potential applications.