EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNN

Area of Science:

• Neuroscience and Artificial Intelligence
• Brain-Computer Interfaces
• Signal Processing

Background:

• Noninvasive brain signal analysis using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) is vital for understanding mental states and neurological conditions.
• EEG offers high temporal resolution, while fNIRS provides better spatial information.
• Multimodal deep learning for EEG-fNIRS analysis is emerging, but subject-independent evaluation is limited.

Purpose of the Study:

• To introduce EF-Net, a CNN-based multimodal deep learning model for analyzing EEG and fNIRS signals.
• To evaluate EF-Net's performance in subject-independent, subject-dependent, and subject-semidependent settings for mental state recognition.
• To demonstrate EF-Net's superiority over traditional and deep learning baseline methods.

Main Methods:

• Developed EF-Net, a novel Convolutional Neural Network (CNN) architecture integrating EEG and fNIRS data.
• Evaluated the model on an EEG-fNIRS word generation dataset for mental state recognition.
• Conducted comparative analysis against five baseline methods (three traditional ML, two DL) across different subject split settings.

Main Results:

• EF-Net achieved superior performance in accuracy and F1 score compared to all baseline models.
• The model demonstrated strong results in subject-dependent (99.36% F1) and subject-semidependent (98.31% F1) settings.
• Significantly, EF-Net attained 65.05% F1 score in the challenging subject-independent setting, outperforming the best baseline by 2.13%.

Conclusions:

• EF-Net effectively learns and interprets mental states from combined EEG and fNIRS signals.
• The model exhibits robust generalization capabilities, performing well on unseen subjects.
• This work advances multimodal brain signal analysis, particularly for subject-independent applications in real-world scenarios.