Real-time mental stress detection using multimodality expressions with a deep learning framework.

Area of Science:

• Computational neuroscience
• Affective computing
• Biomedical engineering

Background:

• Mental stress is a growing societal concern impacting health.
• Timely stress detection is crucial for mitigating adverse effects.
• Existing deep learning stress detection methods often rely on single modalities.

Purpose of the Study:

• To propose a real-time deep learning framework for acute stress detection.
• To integrate multimodal data (ECG, voice, facial expressions) for enhanced accuracy.
• To develop a computer-aided tool for stress detection.

Main Methods:

• A deep learning framework fusing ECG, voice, and facial expression data.
• Utilized ResNet50 and I3D with a temporal attention module (TAM) for feature extraction.
• Employed a matrix eigenvector-based approach for multimodal information fusion.
• Validated using the Montreal Imaging Stress Task (MIST) with 20 participants.

Main Results:

• The proposed framework achieved 85.1% accuracy in acute stress detection.
• Demonstrated the effectiveness of fusing multimodal stress-related information.
• The temporal attention module effectively highlighted stress-specific facial expression features.

Conclusions:

• The multimodal deep learning framework accurately detects acute stress.
• This approach offers a promising tool for computer-aided stress detection.
• Integrating multiple data sources improves stress detection capabilities.