Spatiotemporal multimodal emotion recognition using Temporal video sequences and pose features for child emotion...

Area of Science:

• Developmental Psychology
• Affective Computing
• Computer Vision

Background:

• Accurate identification of children's emotional cues is crucial in developmental psychology and affective computing.
• Existing methods often struggle with the dynamic and subtle nature of children's emotional expressions.
• There is a need for robust systems capable of analyzing multimodal data for reliable emotion recognition.

Purpose of the Study:

• To propose a novel Spatio-Temporal Multimodal Emotion Recognition Network (ST-MERN) for child emotion classification.
• To leverage dense feature embeddings and temporal video sequences for enhanced emotion detection.
• To evaluate the ST-MERN's performance against established models like LSTM and TCN.

Main Methods:

• Utilized the EmoReact dataset with 115 continuous frames per visual signal instance.
• Incorporated rotational-translational vectors, facial keypoints, and pose predictions.
• Captured dynamic data including scale and frame-to-frame variations (r_x, r_y, r_z, t_x, t_y) and latent features (p24-p33).

Main Results:

• The proposed ST-MERN achieved a high validation accuracy of 93.6% and test accuracy of 94.3% using a BiLSTM-based architecture.
• The BiLSTM model demonstrated superior generalization and an F1-score of 0.92.
• The Temporal Convolutional Network (TCN) variant achieved a test accuracy of 91.7% with fast inference times (~0.8s).

Conclusions:

• The ST-MERN effectively captures nuanced emotional expressions in children, outperforming previous models in generalization.
• The system provides interpretable classification sensitive to the dynamic nature of emotional displays.
• This research lays the groundwork for socially sensitive AI systems, therapeutic tools, and educational materials.