A Parallel Multi-Modal Factorized Bilinear Pooling Fusion Method Based on the Semi-Tensor Product for Emotion...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multi-modal fusion leverages complementary data from diverse sources to enhance prediction and classification accuracy.
• Emotion recognition benefits from integrating various data streams, but traditional fusion methods face challenges with dimensionality and redundancy.

Purpose of the Study:

• To propose a novel parallel, multi-modal, factorized, bilinear pooling method using the semi-tensor product (STP) for effective information fusion in emotion recognition.
• To address limitations in fusing modalities with different scales and dimensions, while minimizing data redundancy and computational overhead.

Main Methods:

• The proposed method employs the semi-tensor product (STP) to factorize high-dimensional weight matrices into low-rank matrices, enabling feature projection and interaction capture without dimension matching.
• A parallel, multi-modal, factorized, bilinear pooling approach is utilized, followed by an STP-pooling technique for dimensionality reduction and final feature extraction.

Main Results:

• Experimental validation on IEMOCAP and CMU-MOSI datasets demonstrated significant reductions in storage space and recognition time.
• The method achieved improved performance in emotion recognition, accompanied by decreased training time and a lower number of model parameters.

Conclusions:

• The STP-based multi-modal fusion method effectively integrates information from modalities of varying scales and dimensions.
• This approach offers a computationally efficient and high-performing solution for emotion recognition tasks, reducing resource requirements and enhancing accuracy.