Multimodal approach to situation awareness classification using physiological sensors

Area of Science:

• Human Factors
• Cognitive Science
• Machine Learning

Background:

• Operator situation awareness (SA) is vital for safety and performance.
• Current SA measurement methods often require task interruption, limiting real-world application.
• Developing non-invasive methods to predict SA is a significant research need.

Purpose of the Study:

• To develop and evaluate a multimodal ensemble model for predicting operator situation awareness (SA).
• To assess the effectiveness of non-invasive physiological sensors in forecasting high/low SA levels.
• To identify key physiological indicators contributing to SA prediction.

Main Methods:

• Utilized a dataset of 31 participants performing the Multi-Attribute Task Battery II (MATB-II).
• Employed six non-invasive physiological sensors, including electroencephalogram (EEG) and eye-tracking.
• Developed logistic regression models for individual sensors and combined them using a weighted average ensemble.

Main Results:

• The multimodal ensemble model significantly outperformed baseline methods in predicting high/low SA for levels 1 and 2.
• Electroencephalogram (EEG) and eye-tracking data were identified as the most important sensors for SA prediction.
• Prediction accuracy for level 3 SA was less successful, not significantly outperforming a constant-class baseline.

Conclusions:

• Non-invasive physiological data, particularly EEG and eye-tracking, can effectively predict operator situation awareness (SA) using linear classifiers.
• The developed multimodal ensemble approach offers a promising, non-interruption-based method for SA assessment.
• Further research is needed to improve the prediction of higher-level cognitive states like level 3 SA.