Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving

Area of Science:

• Human-Computer Interaction
• Automotive Safety
• Affective Computing

Background:

• Driver emotion recognition research is growing, but existing datasets lack consistency.
• Current methods often rely on inferred emotional states, leading to inaccuracies.

Purpose of the Study:

• To develop a novel system for collecting multimodal datasets during real-world driving.
• To enable direct, self-reported emotion input from drivers via a Human-Machine Interface (HMI) application.
• To create a reliable foundation for large-scale driver emotion recognition research.

Main Methods:

• A data collection system was designed to minimize driver behavioral and cognitive disturbances.
• Multimodal data was collected during over 122 hours of real-world driving.
• A self-reportable HMI application allowed drivers to input their current emotional state directly.

Main Results:

• A comprehensive multimodal dataset of real-world driving emotions was successfully collected.
• Case studies demonstrated the dataset's utility for statistical analysis, face detection, and personalized emotion recognition.
• The data collection process ensured no accidents occurred and minimized driver disturbance.

Conclusions:

• The proposed system facilitates the creation of reliable, large-scale datasets for driver emotion recognition.
• This work addresses limitations in existing datasets, paving the way for more accurate driver monitoring systems.
• The developed system and dataset are publicly available on GitHub to support further research.