Multimodal Vigilance Estimation With Modality-Pairwise Contrastive Loss.

Area of Science:

• Neuroscience
• Machine Learning
• Biomedical Engineering

Background:

• Estimating human vigilance accurately is crucial but challenged by complex cross-modal interactions in multimodal fusion.
• Existing methods struggle with effectively integrating diverse data sources like EEG and EOG for vigilance monitoring.

Purpose of the Study:

• To propose a novel cross-modality alignment method using contrastive learning for improved multimodal vigilance estimation.
• To extract shared semantic information across modalities while minimizing intermodal differences.

Main Methods:

• A contrastive learning framework was developed to align representations from different modalities (EEG, EOG).
• The method maximizes the similarity of semantic representations to reduce intermodal discrepancies.
• The approach was evaluated on the SEED-VIG dataset for both intra-subject and inter-subject vigilance estimation.

Main Results:

• Achieved state-of-the-art performance in multimodal vigilance estimation, with improved RMSE/CORR of 0.092/0.893 (intra-subject) and 0.144/0.887 (inter-subject).
• Identified theta and alpha brainwave activities as key indicators for vigilance estimation.
• Demonstrated that contrastive learning significantly enhances the correlation between brain activity and PERCLOS.

Conclusions:

• The proposed contrastive learning method effectively addresses challenges in multimodal fusion for vigilance estimation.
• The approach shows potential for reducing high-cost data annotation in inter-subject scenarios.
• Findings suggest avenues for multimodal vigilance regression applications.