$\ell _{0}$ℓ0-Regularized Sparse Coding-Based Interpretable Network for Multi-Modal Image Fusion.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Multi-modal image fusion (MMIF) combines information from different sensor images to improve visualization and detection.
• Existing methods often lack interpretability and can be computationally intensive.

Purpose of the Study:

• To introduce FNet, an interpretable network for MMIF based on a novel $\ell _{0}$-regularized multi-modal convolutional sparse coding (MCSC) model.
• To develop a learnable $\ell _{0}$-regularized sparse coding (LZSC) block using deep unfolding for efficient feature extraction.
• To propose an interpretable inverse fusion network (IFNet) for improved training.

Main Methods:

• FNet utilizes a deep unfolding approach to implement the LZSC block for separating unique and common features.
• The MCSC model is adapted for the inverse fusion process, enabling the IFNet.
• The networks are trained and evaluated on eight diverse MMIF datasets.

Main Results:

• FNet achieved high-quality fusion results across multiple datasets.
• The fused images from FNet demonstrated improved performance in downstream object detection and semantic segmentation tasks, particularly with visible-thermal image pairs.
• Visualization of FNet's intermediate results confirmed its interpretability.

Conclusions:

• FNet offers an effective and interpretable solution for multi-modal image fusion.
• The proposed LZSC block and IFNet contribute to advancements in sparse coding-based image fusion.
• The approach shows promise for enhancing various computer vision applications requiring fused image data.