Recognition of emotions using multimodal physiological signals and an ensemble deep learning model

Area of Science:

• Affective computing
• Machine learning
• Physiological signal processing

Background:

• Deep learning for emotion recognition from physiological signals is gaining traction.
• Conventional deep classifiers struggle with model structure determination and multimodal feature fusion.

Purpose of the Study:

• To propose a Multiple-fusion-layer based Ensemble classifier of Stacked Autoencoder (MESAE) for improved emotion recognition.
• To address limitations in model structure and feature abstraction in existing deep emotion classifiers.

Main Methods:

• A physiological-data-driven approach identifies the deep structure of the MESAE.
• Stacked Autoencoders (SAEs) with three hidden layers extract stable feature representations from physiological signals.
• A three-layer, adjacent-graph-based network fuses SAE abstractions for binary arousal/valence state recognition.

Main Results:

• The MESAE was validated using the DEAP multimodal database.
• Compared to the best existing emotion classifiers, MESAE achieved a 5.26% improvement in mean classification rate and F-score.

Conclusions:

• The MESAE demonstrates superior performance over state-of-the-art shallow and deep emotion classifiers.
• The model's effectiveness is shown across varying numbers of physiological data instances.