PMMCT: A Parallel Multimodal CNN-Transformer Model to Detect Slow Eye Movement for Recognizing Driver Sleepiness

Area of Science:

• Neuroscience
• Computer Science
• Transportation Safety

Background:

• Sleepiness at the wheel is a significant cause of road traffic accidents.
• Slow eye movement (SEM) is a reliable physiological indicator of the sleep onset period (SOP).
• Accurate detection of SOP is vital for preventing drowsy driving incidents.

Purpose of the Study:

• To propose and evaluate a novel Parallel Multimodal CNN-Transformer (PMMCT) model for detecting SEM and recognizing drivers' SOP.
• To assess the performance of the PMMCT model using bimodal signal combinations.
• To compare the PMMCT model's efficacy against existing deep learning architectures.

Main Methods:

• Development of a PMMCT model with parallel feature extraction modules using convolutional layers and Transformer encoder layers.
• Processing of bimodal signals, specifically HEOG + O2 and HEOG + HSUM (a novel combination capturing EEG-like features).
• Fusion of extracted features and classification using fully connected layers.

Main Results:

• The PMMCT model achieved high performance, with average classification accuracy of 99.89% and F1-score of 99.35%.
• The HEOG + HSUM combination performed comparably to HEOG + O2 and outperformed unimodal HEOG by 2.73% in F1-score.
• The model demonstrated minimal false positives (5.2) and false negatives (0.8), outperforming CNN, CNN-LSTM, and CNN-LSTM-Attention models.

Conclusions:

• The PMMCT model effectively detects SEM for recognizing drivers' SOP, combining CNNs' local feature extraction with Transformers' global temporal modeling.
• The HEOG + HSUM bimodal combination offers a promising alternative for SOP detection, potentially reducing the need for more complex EEG setups.
• The system provides superior performance with enhanced comfort for wearable applications, contributing to improved road safety.