Probing Synergistic High-Order Interaction for Multi-Modal Image Fusion

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Multi-modal image fusion integrates complementary information from multiple sources.
• Existing cross-attention mechanisms capture only second-order spatial interactions, limiting synergy exploitation.
• Higher-order interactions in spatial and channel dimensions are underexplored.

Purpose of the Study:

• To introduce a novel Synergistic High-order Interaction Paradigm (SHIP) for multi-modal image fusion.
• To address the limitations of second-order interactions in current fusion models.
• To enhance the exploitation of multi-modal synergies through high-order statistics.

Main Methods:

• Developed SHIP to systematically investigate spatial and channel dimension collaborations.
• Implemented element-wise multiplication for fine-grained spatial interactions, equivalent to global interactions.
• Devised high-order channel interactions using global statistics to discern inter-dependencies.
• Introduced SHIP++ with enhanced cross-modality interaction representation via cross-order attention and residual memorization.

Main Results:

• SHIP effectively captures high-order spatial and channel interactions.
• SHIP++ demonstrates superior performance over state-of-the-art methods in multi-modal fusion tasks.
• Experiments on pan-sharpening and infrared/visible image fusion validate the model's efficacy.

Conclusions:

• High-order interactions significantly boost the ability to exploit multi-modal synergies.
• The proposed SHIP paradigm offers a flexible and efficient approach to advanced image fusion.
• SHIP and SHIP++ represent a significant advancement in multi-modal image fusion techniques.