Enhancing Multimodal Learning via Hierarchical Fusion Architecture Search With Inconsistency Mitigation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multimodal learning requires effective feature fusion strategies, often demanding significant computational resources and expertise.
• Existing methods lack mechanisms to address inconsistencies between modalities and labels during fusion architecture design.

Purpose of the Study:

• To develop an efficient Hierarchical Fusion Multimodal Neural Architecture Search (HF-MNAS) method.
• To mitigate modality-label inconsistencies in multimodal feature fusion.
• To reduce the computational cost associated with designing fusion architectures.

Main Methods:

• Introduced a two-level search space: macro-level for feature extraction and connection, and micro-level for cell optimization.
• Developed an inconsistency mitigation module to minimize discrepancies between modalities and labels.
• Implemented an importance-based node selection mechanism for optimal cell formation.

Main Results:

• HF-MNAS achieved a competitive balance between accuracy, search time, and inference speed on multimodal classification tasks.
• Demonstrated significantly lower computational costs compared to state-of-the-art methods.
• Verified that modality-label inconsistency negatively impacts model performance and that the proposed module effectively mitigates this.

Conclusions:

• HF-MNAS offers an efficient and effective approach to multimodal feature fusion architecture search.
• Addressing modality-label inconsistency is crucial for improving multimodal learning performance.
• The proposed method provides a practical solution for resource-constrained multimodal learning applications.