Multi-Branch Tree-Based Fusion Neural Architecture Search With Zero-Cost Screen for Multi-Modal Classification

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Multi-modal classification (MMC) integrates diverse data for improved performance.
• Current fusion methods require significant expertise and resources, lacking flexibility.
• Existing neural architecture search (NAS) methods for fusion are computationally intensive and limited in capturing complex correlations.

Purpose of the Study:

• To develop an efficient and accurate multi-modal fusion framework.
• To overcome the limitations of expert-designed architectures and computationally expensive NAS methods.
• To enhance cross-modal interaction and modality importance weighting.

Main Methods:

• Proposed a multi-branch tree-based fusion neural architecture search (MBTF-NAS) framework.
• Employed a multi-branch tree-structured encoding for dynamic fusion topology exploration.
• Integrated a learnable model-level attention weighting mechanism.
• Utilized zero-cost proxy metrics for efficient architecture evaluation.

Main Results:

• MBTF-NAS demonstrated superior performance across seven multi-modal benchmarks.
• Achieved high accuracy and efficiency in multi-modal fusion.
• Effectively strengthened cross-modal interactions and modality emphasis.

Conclusions:

• MBTF-NAS offers an effective and generalizable solution for multi-modal classification.
• The framework successfully balances accuracy and computational efficiency.
• Represents a significant advancement over existing state-of-the-art approaches.