Deep kernel learning enhanced fusion for multimodal classification.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multimodal learning integrates diverse data types into a unified representation.
• Current methods often suffer from feature-level bias due to varying modality informativeness.
• This bias introduces noise, hindering the performance of multimodal classification tasks.

Purpose of the Study:

• To propose a novel Deep Kernel Learning enhanced Multimodal Classification (DKLMC) framework.
• To address the feature-level bias problem in multimodal learning.
• To enhance the accuracy and robustness of multimodal classification.

Main Methods:

• Developed a DKLMC framework with a main classification network and a deep kernel learning auxiliary network.
• The auxiliary network estimates unimodal contribution factors, quantifying modality informativeness.
• Utilized the Reptile algorithm for meta-learning to balance optimization between networks.

Main Results:

• The DKLMC framework effectively estimates the contribution of each modality per instance.
• It emphasizes informative features while downplaying less relevant ones, mitigating feature-level bias.
• Experiments across diverse datasets demonstrated significant improvements in classification performance.

Conclusions:

• The proposed DKLMC framework offers an effective solution to feature-level bias in multimodal learning.
• Deep kernel learning and meta-learning integration enhance modality contribution estimation and classification accuracy.
• DKLMC shows strong adaptability and effectiveness across various multimodal tasks.