DM-Fusion: Deep Model-Driven Network for Heterogeneous Image Fusion

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Deep neural network-based heterogeneous image fusion (HIF) methods often lack theoretical guarantees and optimal convergence.
• Existing convolutional neural network approaches in HIF function as black boxes, limiting interpretability.

Purpose of the Study:

• To develop a novel deep model-driven neural network for heterogeneous image fusion (HIF).
• To integrate model-based interpretability with deep learning generalizability for enhanced HIF.
• To create a compact, explainable, and effective HIF network.

Main Methods:

• Designed a deep model-driven neural network (DM-fusion) integrating domain knowledge with deep learning.
• Developed a tailored objective function and domain knowledge network modules for interpretability.
• Implemented an iterative parameter learning scheme and a task-driven loss function for feature enhancement and preservation.

Main Results:

• DM-fusion demonstrated feasibility and effectiveness across three key components: HIF model, parameter learning, and network architecture.
• Experiments on four fusion tasks and downstream applications showed DM-fusion outperforms state-of-the-art methods.
• The proposed method achieved advancements in both fusion quality and computational efficiency.

Conclusions:

• The deep model-driven approach offers a promising direction for heterogeneous image fusion.
• DM-fusion provides a more interpretable and efficient solution compared to purely data-driven methods.
• The developed technique shows significant potential for various image fusion applications.