Spatial-Frequency Enhanced Mamba for Multi-Modal Image Fusion

Area of Science:

• Computer Vision
• Artificial Intelligence
• Signal Processing

Background:

• Multi-Modal Image Fusion (MMIF) integrates complementary data from diverse sources.
• Existing deep learning methods like CNNs and Transformers have limitations in receptive field and computational cost for MMIF.
• Mamba shows promise for long-range dependencies but lacks spatial and frequency perception crucial for MMIF.

Purpose of the Study:

• To propose a novel framework, Spatial-Frequency Enhanced Mamba Fusion (SFMFusion), for improved MMIF.
• To address the limitations of existing methods by enhancing Mamba's capabilities for spatial and frequency perception.
• To effectively leverage Image Reconstruction (IR) as an auxiliary task within the MMIF framework.

Main Methods:

• Developed a three-branch structure to couple MMIF and IR, preserving complete source image content.
• Introduced the Spatial-Frequency Enhanced Mamba Block (SFMB) to augment Mamba with comprehensive spatial and frequency domain feature extraction.
• Proposed the Dynamic Fusion Mamba Block (DFMB) for adaptable feature fusion across different network branches.

Main Results:

• SFMFusion achieved superior performance compared to state-of-the-art methods on six MMIF datasets.
• The proposed SFMB and DFMB modules effectively enhanced feature extraction and fusion capabilities.
• The integrated approach demonstrated robust performance in multi-modal image fusion tasks.

Conclusions:

• SFMFusion offers a significant advancement in MMIF by effectively integrating Mamba with spatial and frequency enhancements.
• The framework successfully addresses limitations of previous deep learning approaches.
• The method provides a promising direction for future research in image fusion and related AI applications.