EFRM: A Multimodal EEG-fNIRS Representation-learning Model for few-shot brain-signal classification.

Area of Science:

• Neuroscience
• Machine Learning
• Biomedical Engineering

Background:

• Robust brain signal classifiers are needed for minimal labeled data scenarios.
• Transfer learning is a promising strategy, but existing methods often lack generalization across modalities like EEG and fNIRS.

Purpose of the Study:

• To develop a multimodal representation model adaptable to EEG-only, fNIRS-only, and paired EEG-fNIRS datasets.
• To improve transfer learning performance for brain signal classification, particularly for fNIRS signals.

Main Methods:

• A two-stage approach: pre-training for modality-specific and shared representations, followed by transfer learning for downstream tasks.
• Utilized large-scale unlabeled datasets (approx. 1250 h from 918 participants) for pre-training.
• Enabled adaptation to single-modality datasets, unlike previous multimodal methods.

Main Results:

• The multimodal model outperformed single-modality approaches by leveraging shared domains across EEG and fNIRS.
• Achieved competitive performance against state-of-the-art supervised models, even with limited labeled data.
• Demonstrated significant improvements in fNIRS classification compared to previously pre-trained models.

Conclusions:

• The proposed method offers a flexible and practical solution for brain signal classification using transfer learning.
• Effective for both single and multimodal brain signal data, enhancing classifier robustness.
• Advances the application of transfer learning in neuroimaging analysis, especially for fNIRS.