Integrating audio and visual modalities for multimodal personality trait recognition via hybrid deep learning

Area of Science:

• Artificial Intelligence
• Psychology
• Affective Neuroscience

Background:

• Personality trait recognition analyzes psychological characteristics from behavior data.
• Leveraging audio-visual cues is crucial for understanding first impressions.

Purpose of the Study:

• To propose a novel multimodal personality trait recognition method.
• To integrate audio-visual modalities using a hybrid deep learning framework.

Main Methods:

• Utilized pre-trained Convolutional Neural Networks (CNN) for audio and face feature extraction.
• Employed Bi-directional Long Short-Term Memory (Bi-LSTM) and Transformer networks to capture temporal dependencies.
• Integrated audio-visual features using a decision-level fusion strategy with linear regression.

Main Results:

• The proposed hybrid deep learning framework effectively recognized personality traits.
• The method demonstrated superior performance on the ChaLearn First Impression-V2 dataset.
• Audio-visual integration outperformed single-modality approaches.

Conclusions:

• The developed method offers an effective approach for multimodal personality trait recognition.
• The integration of audio-visual data significantly enhances the accuracy of personality assessment.
• This research advances the application of AI in understanding human psychological characteristics.