Real-World Driver Stress Recognition and Diagnosis Based on Multimodal Deep Learning and Fuzzy EDAS Approaches

Area of Science:

• Engineering
• Computer Science
• Psychology

Background:

• Mental stress is a significant factor in road accidents and can lead to various health disorders.
• Previous research relied on manual feature engineering from physiological, physical, and contextual data, which is challenging.
• Deep learning (DL) offers automated feature extraction, simplifying stress recognition model development.

Purpose of the Study:

• To propose and evaluate Convolutional Neural Network (CNN) and CNN-Long Short-Term Memory (LSTM) fusion models for driver stress detection.
• To utilize physiological signals and multimodal data for recognizing two and three levels of driver stress.
• To assess model performance using the fuzzy Evaluation Based on Distance from Average Solution (EDAS) approach.

Main Methods:

• Development of CNN and hybrid CNN-LSTM models for stress recognition.
• Application of models to the SRAD dataset (physiological signals) and AffectiveROAD dataset (multimodal data).
• Performance evaluation using fuzzy EDAS with metrics including accuracy, recall, precision, F-score, and specificity.

Main Results:

• The proposed CNN and CNN-LSTM models ranked highest when fusing data from Bio harnesses (BH), E4-Left (E4-L), and E4-Right (E4-R) sensors.
• Multimodal data significantly improved the accuracy and trustworthiness of the driver stress recognition model.
• The models demonstrated effectiveness in identifying two and three distinct levels of driver stress.

Conclusions:

• Deep learning-based fusion models, particularly CNN and CNN-LSTM, are effective for automated driver stress recognition.
• The integration of multimodal data is crucial for developing robust and reliable stress detection systems for driving.
• The developed models show potential for diagnosing stress levels in various daily activities beyond driving.