Multimodal Approach for Emotion Recognition Based on Simulated Flight Experiments

Area of Science:

• Human-Computer Interaction
• Affective Computing
• Aerospace Psychology

Background:

• Understanding pilot emotions is crucial for flight safety and performance.
• Previous research has gaps in correlating physiological responses with specific flight phases.
• Simulated flight environments offer a controlled setting for studying pilot emotional states.

Purpose of the Study:

• To develop a sensing architecture for capturing pilot bio-reactions during simulated flights.
• To recognize pilot emotions (happy, sad, angry, surprise, scared) using physiological and facial data.
• To evaluate the effectiveness of Artificial Neural Networks and Deep Learning in emotion recognition.

Main Methods:

• Utilized Microsoft Flight Simulator (FSX-SE) for 13 simulated flights with 8 beginner pilots.
• Collected physiological data: Heart Rate (HR), Galvanic Skin Response (GSR), and Electroencephalography (EEG).
• Employed Artificial Neural Networks and Deep Learning for emotion classification, measuring accuracy with Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE).

Main Results:

• Emotion recognition models performed best when using all data or omitting GSR data.
• The emotion 'surprised' was most accurately recognized (mean RMSE 0.13, MAE 0.01).
• The emotion 'sad' was least accurately recognized (mean RMSE 0.82, MAE 0.08), with accuracies between 55%-100% for high-intensity emotions.

Conclusions:

• It is feasible to recognize pilot emotions during flight by integrating present and past emotional states.
• Physiological signals (HR, GSR, EEG) combined with facial analysis provide valuable insights into pilot affect.
• The findings contribute to developing more responsive and adaptive flight training and monitoring systems.