Multimodal Vigilance Estimation Using Deep Learning.

Area of Science:

• Neuroscience
• Transportation Safety
• Machine Learning

Background:

• Reduced vigilance is a growing cause of accidents, particularly in public transportation.
• Accurate estimation of vigilance is essential for enhancing safety in transportation systems.
• Current methods for vigilance estimation require improvement in accuracy and reliability.

Purpose of the Study:

• To propose and evaluate a novel multimodal regression network for accurate vigilance estimation.
• To assess the effectiveness of the proposed network using electroencephalography and electrooculography features.
• To demonstrate the benefits of feature fusion for improved vigilance state classification.

Main Methods:

• Development of a multimodal regression network: multichannel deep autoencoders with subnetwork neurons (MCDAEsn).
• Utilizing forehead electrooculography (fEEG) and electroencephalography (EEG) as input modalities.
• Implementing feature fusion of fEEG and EEG data to capture complementary information.

Main Results:

• The MCDAEsn model achieved high accuracy in distinguishing between awake, tired, and drowsy states based on eye closure percentages (0-0.35, 0.36-0.70, 0.71-1).
• Single modality application showed promising results, validating the core network architecture.
• Multimodal fusion of fEEG and EEG features significantly enhanced the performance compared to single modalities, demonstrating method effectiveness.

Conclusions:

• The proposed MCDAEsn network offers an effective and efficient solution for accurate vigilance estimation.
• Multimodal data fusion, particularly combining fEEG and EEG, substantially improves the performance of vigilance monitoring systems.
• This approach holds significant potential for enhancing safety in public transportation by proactively identifying and mitigating risks associated with reduced vigilance.