A Dual-Branch Lightweight Network for Multimodal Image Fusion with Mamba and INN.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Multimodal image fusion integrates data from diverse imaging sources.
• Current deep learning fusion methods are computationally intensive, limiting real-time use.
• Resource-constrained devices require efficient fusion techniques.

Purpose of the Study:

• To develop a lightweight deep learning model for efficient multimodal image fusion.
• To address the high computational cost of existing fusion methods.
• To enable real-time image fusion on resource-limited hardware.

Main Methods:

• A dual-branch network combining a Mamba-inspired module for global context and an Invertible Neural Network (INN) for local details.
• The INN branch utilizes reversible transformations to preserve high-frequency textures and edge information.
• Techniques like shallow feature refinement, module reuse, and a streamlined decoder minimize computational overhead.

Main Results:

• The proposed Mamba-INN network achieves competitive fusion quality on infrared-visible and medical image datasets (MSRS, TNO, RoadScene, MRI-CT, MRI-PET, MRI-SPECT).
• It demonstrates performance comparable to or exceeding methods like CDDFuse and U2Fusion across various metrics (MI, VIF, Qabf, SSIM).
• The model boasts significantly reduced complexity (0.24M parameters, 24.04 GFLOPs at 256x256), enabling efficient deployment.

Conclusions:

• The lightweight Mamba-INN network offers an efficient solution for multimodal image fusion.
• It successfully balances fusion quality with significantly reduced computational complexity.
• The method shows strong potential for real-time applications on resource-constrained devices.