Pre-gating and contextual attention gate - A new fusion method for multi-modal data tasks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multi-modal representation learning is crucial for comprehensive scenario modeling.
• Effective cross-modal interaction learning is key for joint data representation.
• Conventional cross-attention can introduce noise and uncertainty, degrading performance.

Purpose of the Study:

• To introduce a novel Pre-gating and Contextual Attention Gate (PCAG) module for enhanced multi-modal learning.
• To address the limitations of conventional cross-attention mechanisms in multi-modal fusion.

Main Methods:

• Developed a PCAG module with two distinct gating mechanisms.
• The first gate filters uninformative cross-modal interactions.
• The second gate mitigates uncertainty from cross-attention modules.

Main Results:

• The PCAG module significantly improved multi-modal fusion model performance across eight diverse classification tasks.
• Outperformed existing state-of-the-art multi-modal fusion models.
• Demonstrated effective processing of cross-modality interactions.

Conclusions:

• The PCAG module offers a robust solution for enhancing multi-modal representation learning.
• PCAG effectively filters noise and reduces uncertainty, leading to superior performance in downstream tasks.