Multi-Scale Temporal Fusion Network for Real-Time Multimodal Emotion Recognition in IoT Environments

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Signal Processing

Background:

• Emotion recognition in the Internet of Things (IoT) faces challenges with diverse sensor data and temporal scales.
• Existing methods struggle to balance accuracy, latency, and energy efficiency on resource-constrained IoT devices.
• Processing multi-modal sensor data (physiological, visual, audio) requires sophisticated fusion techniques.

Purpose of the Study:

• To introduce EmotionTFN, a novel hierarchical temporal fusion network for accurate IoT emotion recognition.
• To enable efficient deployment of emotion recognition systems on edge devices.
• To validate the performance and real-world applicability of the proposed architecture.

Main Methods:

• Developed Emotion-Multi-Scale Temporal Fusion Network (EmotionTFN) with hierarchical temporal attention.
• Integrated physiological (EEG, PPG, GSR), visual, and audio data across short, medium, and long-term temporal windows.
• Applied edge computing optimizations: model compression, quantization, and adaptive sampling.

Main Results:

• Achieved 94.2% accuracy in discrete emotion classification and 0.087 mean absolute error in dimensional prediction on MELD, DEAP, and G-REx datasets.
• Demonstrated sub-200 ms latency on IoT hardware with a 40% improvement in energy efficiency.
• Validated real-world deployment with 97.2% uptime and high user satisfaction (4.1/5.0).

Conclusions:

• EmotionTFN effectively processes multi-modal sensor data for IoT emotion recognition across multiple temporal scales.
• Edge computing optimizations enable efficient and accurate emotion recognition on resource-constrained devices.
• The system ensures user privacy through local data processing while maintaining high performance and reliability.