POF cardiorespiratory sensor combined with a thermal imaging temperature sensor for multimodal emotion recognition

Area of Science:

• Physiological computing
• Affective computing
• Biomedical engineering

Background:

• Emotion recognition is crucial for driver monitoring and mental health assessment.
• Current expression-based methods lack individual reliability.
• Physiological signals offer a more objective approach, but single modalities are insufficient for comprehensive emotional state representation.

Purpose of the Study:

• To develop and evaluate a multimodal sensing approach for enhanced emotion recognition.
• To integrate cardiorespiratory signals and facial thermal data for a more robust assessment.
• To compare multimodal fusion performance against single-modality baselines.

Main Methods:

• A polymer optical fiber (POF) cardiorespiratory sensor was used to capture thoracic expansion and derive cardiorespiratory signals.
• A thermal image temperature sensor provided high-sensitivity infrared measurements of facial temperature.
• Features from both modalities were extracted, fused into a 42-dimensional vector, and classified using SVM, KNN, and RF within a nested cross-validation framework.

Main Results:

• Multimodal fusion significantly reduced classification error compared to single-modality methods.
• Peak classification accuracy of 93% was achieved using Support Vector Machine (SVM) with feature selection.
• The fused feature vector effectively represented physiological patterns during induced emotional fluctuations.

Conclusions:

• Integrating portable POF cardiorespiratory sensing with thermal imaging provides a robust and generalizable method for emotion recognition.
• This multimodal approach overcomes the limitations of single-modality sensing for comprehensive emotional state assessment.
• The findings support the application of this technology in driver-state assessment and mental-health monitoring.