Multi-modal Approach for Affective Computing.

Area of Science:

• Affective Computing
• Human-Computer Interaction
• Biomedical Signal Processing

Background:

• Previous emotion classification studies relied on single sensing modalities (e.g., face video, EEG, ECG, GSR), each with inherent limitations.
• Limited research exists on comparing the effectiveness of individual modalities and their joint application for emotion recognition.
• Existing methods face challenges like missing physiological biomarkers in video or poor resolution in biosignals.

Purpose of the Study:

• To compare the performance of various computational approaches for human emotion classification using single and multi-modal data.
• To investigate the efficacy of combining face video with biosensing modalities (EEG, ECG, GSR) for improved emotion recognition.
• To introduce a novel method for compensating physiological baselines to enhance classification accuracy.

Main Methods:

• Utilized the multi-modal AMIGOS dataset for emotion classification tasks.
• Applied multiple computational approaches to analyze face videos and biosensing data (EEG, ECG, GSR).
• Developed and implemented a novel technique for physiological baseline compensation.

Main Results:

• Demonstrated that multi-modal approaches outperform single-modality methods in human emotion classification.
• The novel baseline compensation method led to a significant increase in classification accuracy across various approaches.
• Identified optimal combinations of modalities and computational methods for robust emotion recognition.

Conclusions:

• Multi-modal sensing, particularly when combined with advanced signal processing techniques, offers superior performance for human emotion classification.
• The proposed physiological baseline compensation method is effective in improving the accuracy of emotion recognition systems.
• This research contributes a robust multi-modal framework for affective computing, enabling more nuanced understanding of human emotions.