A Multimodal Pain Sentiment Analysis System Using Ensembled Deep Learning Approaches for IoT-Enabled Healthcare...

Area of Science:

• Artificial Intelligence
• Healthcare Technology
• Biomedical Signal Processing

Background:

• Unimodal systems for pain sentiment analysis have limitations in accuracy and real-time application.
• Accurate pain assessment is crucial for effective patient care and treatment decisions.
• Internet of Things (IoT)-enabled healthcare frameworks require advanced tools for patient monitoring.

Purpose of the Study:

• To develop and evaluate a multimodal sentiment analysis system for human pain recognition.
• To integrate facial expression and speech-audio data for enhanced pain intensity assessment.
• To improve real-time patient care decision-making in IoT healthcare settings.

Main Methods:

• Facial region detection and feature extraction using deep learning (CNNs) with transfer learning.
• Speech-audio preprocessing and feature extraction, including deep learning for divergent features.
• Fusion of image-based and audio-based pain sentiment analysis model outcomes.
• Implementation across five key phases from detection to final multimodal fusion.

Main Results:

• The multimodal system achieved high accuracies: 99.31% (2-class), 99.54% (3-class), and 87.41% (5-class).
• Individual unimodal systems showed strong performance: image-based up to 84.27%, audio-based up to 98.32%.
• The proposed system demonstrated superiority over state-of-the-art methods in pain sentiment analysis.

Conclusions:

• Multimodal sentiment analysis integrating facial and audio data significantly enhances pain recognition accuracy.
• The developed system offers a robust solution for objective pain assessment in healthcare.
• This technology supports improved patient management and personalized treatment within IoT healthcare frameworks.