A dual-stream feature decomposition network with weight transformation for multi-modality image fusion

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Multi-modal image fusion aims to integrate information from diverse sources into a single image.
• Existing methods using Convolutional Neural Networks (CNNs) have limited receptive fields, while Transformer-based methods are computationally intensive.
• Current approaches often fail to adequately explore cross-domain information and handle feature extraction across different frequencies.

Purpose of the Study:

• To propose an innovative image fusion model for multi-modal images, including infrared/visible and medical imaging pairs.
• To leverage the complementary strengths of Transformers and CNNs for effective modeling of various feature types and learning ranges.
• To enhance downstream visual tasks by generating fused images with preserved salient information and complementary features.

Main Methods:

• A shared Transformer-based encoder for long-range learning, incorporating intra-modal, inter-modal, and feature alignment blocks.
• A private CNN-based dual-stream encoder for low- and high-frequency feature extraction, featuring a dual-domain selection mechanism and an invertible neural network.
• A cross-attention-based Swin Transformer block with embedded weight transformation for efficient cross-domain information exploration and a unified loss function with dynamic weighting.

Main Results:

• The proposed model effectively preserves thermal targets and background texture details in fused images.
• Qualitative and quantitative analyses demonstrate superior performance compared to state-of-the-art methods in image fusion.
• Significant improvements were observed in the performance of subsequent visual tasks, such as object detection.

Conclusions:

• The developed multi-modal image fusion model successfully integrates features from diverse sources using a hybrid Transformer-CNN architecture.
• The model addresses limitations of existing methods by enhancing feature extraction, cross-domain information exploration, and efficiency.
• The proposed approach offers a robust solution for high-quality image fusion, benefiting various downstream computer vision applications.