BC-PMJRS: A Brain Computing-inspired Predefined Multimodal Joint Representation Spaces for enhanced cross-modal...

Area of Science:

• Multimodal Machine Learning
• Computational Neuroscience
• Artificial Intelligence

Background:

• Multimodal learning struggles with fusing diverse data and efficient cross-modal interactions.
• Existing methods often lack biologically plausible mechanisms for complex information processing.

Purpose of the Study:

• To propose BC-PMJRS, a novel Brain Computing-inspired Predefined Multimodal Joint Representation Spaces method.
• To enhance cross-modal learning by integrating principles of neural plasticity and selective attention.
• To improve sentiment analysis performance using multimodal data.

Main Methods:

• Learns joint representations by minimizing inter-modal redundancy and maximizing task-specific discrimination via mutual information.
• Employs an adaptive optimization strategy inspired by long-term potentiation (LTP) and long-term depression (LTD).
• Utilizes a global-local cross-modal interaction mechanism for reduced computational complexity, mimicking selective attention.

Main Results:

• BC-PMJRS outperforms state-of-the-art models on IEMOCAP, MOSI, and MOSEI datasets in both complete and incomplete modality scenarios.
• Achieved up to 1.9% improvement in weighted-F1 on IEMOCAP, 2.8% gain in 7-class accuracy on MOSI, and 2.9% increase on MOSEI.
• Demonstrated significant performance gains, validating the effectiveness of brain-inspired mechanisms.

Conclusions:

• Brain-inspired mechanisms, particularly dynamic balancing of information through neural plasticity, significantly enhance multimodal learning.
• The proposed BC-PMJRS method offers a more effective and biologically plausible approach to multimodal machine learning.
• This research bridges neuroscience and AI, paving the way for advanced multimodal models.