A Multimodal Feature Fusion Framework for Sleep-Deprived Fatigue Detection to Prevent Accidents

Area of Science:

• Human-Computer Interaction
• Biomedical Engineering
• Cognitive Science

Background:

• Sleep deprivation significantly impairs cognitive function and increases error rates.
• Accurate and timely fatigue detection is essential for safety in various critical domains.
• Existing fatigue detection methods are often intrusive or lack comprehensive feature integration.

Purpose of the Study:

• To develop and validate a novel, nonintrusive fatigue detection system.
• To explore the efficacy of multimodal feature fusion for enhanced fatigue recognition.
• To compare the performance of the proposed method against established machine learning classifiers.

Main Methods:

• A multimodal approach was employed, integrating features from visual, thermal, keystroke dynamics, and voice data.
• Empirical weights were assigned to features from the four distinct domains.
• Data was collected from 60 healthy young volunteers (20-30 years) under controlled conditions.
• The proposed method was evaluated against k-nearest neighbors (kNN), support vector machines (SVM), random tree, random forest, and multilayer perceptron classifiers using 3-fold cross-validation.

Main Results:

• The multimodal fatigue detection system achieved an average accuracy of 93.33% in 3-fold cross-validation.
• The proposed technique demonstrated superior performance compared to individual feature domains and several standard classifiers.
• Feature fusion effectively enhanced the robustness and accuracy of fatigue detection.

Conclusions:

• The developed nonintrusive, multimodal fatigue detection system is highly accurate and effective.
• Multimodal feature fusion offers a promising direction for robust fatigue monitoring.
• This approach has significant potential for application in safety-critical environments to mitigate risks associated with fatigue.