CIRF: Coupled Image Reconstruction and Fusion Strategy for Deep Learning Based Multi-Modal Image Fusion.

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Multi-modal medical image fusion (MMIF) is vital for diagnostics, leveraging complementary sensor data.
• Deep learning (DL) methods are prevalent but suffer from serial fusion, leading to errors and scale confusion.
• Existing DL approaches lack feature decomposition, hindering performance in MMIF.

Purpose of the Study:

• To propose a novel Coupled Image Reconstruction and Fusion (CIRF) strategy for enhanced MMIF.
• To address limitations of serial fusion and error accumulation in DL-based MMIF.
• To improve the expressivity and accuracy of fused medical images.

Main Methods:

• Developed the CIRF strategy with parallel image fusion and reconstruction branches.
• Employed a common encoder with Vision Transformer (ViT) and Convolutional Neural Network (CNN) branches for feature extraction.
• Integrated multi-task learning with supervised and unsupervised losses, incorporating feature decomposition.

Main Results:

• CIRF demonstrated superior performance in subjective and objective evaluations across three datasets.
• Fused images exhibited appropriate brightness and smooth edge transitions.
• CIRF achieved more competitive evaluation metrics compared to traditional and other DL-based methods.

Conclusions:

• The proposed CIRF strategy effectively enhances MMIF by combining parallel processing, feature decomposition, and multi-task learning.
• CIRF overcomes limitations of serial fusion, offering improved accuracy and image quality.
• The method shows strong generalization capabilities and potential for clinical applications.