Multimodal driver emotion recognition using motor activity and facial expressions.

Area of Science:

• Human-Computer Interaction
• Affective Computing
• Transportation Safety

Background:

• Intense emotions (anger, sadness, agitation, joy) significantly impact driving performance and increase accident risk.
• Existing methods for emotion recognition in drivers are limited in scope and accuracy.
• Understanding driver emotional states is crucial for developing advanced driver-assistance systems and improving road safety.

Purpose of the Study:

• To develop a multimodal intelligent model for recognizing four specific driver emotions: anger, sadness, agitation, and joy.
• To integrate motor activity signals and facial geometry images for accurate emotion classification.
• To investigate the relationship between driver behavior, motor activity, facial geometry, and induced emotions.

Main Methods:

• Utilized machine learning to identify key motor activity signals relevant to emotion recognition.
• Employed a pre-trained Convolutional Neural Network (CNN) to extract probability vectors from facial geometry images.
• Integrated motor activity and facial data using a unidimensional network for multimodal emotion classification.

Main Results:

• The multimodal intelligent model achieved 96.0% accuracy in recognizing four specific emotions in a simulated driving environment.
• A significant correlation was confirmed between drivers' motor activity, behavior, facial geometry, and the induced emotional states.
• Motor activity signals and facial geometry proved to be effective modalities for driver emotion detection.

Conclusions:

• The developed multimodal model offers a highly accurate approach to recognizing driver emotions.
• This technology has the potential to enhance driver monitoring systems and contribute to safer driving.
• Further research can explore real-world applications and the recognition of a wider range of emotions.