Deep M²CDL: Deep Multi-Scale Multi-Modal Convolutional Dictionary Learning Network.

Area of Science:

• Computer Vision
• Machine Learning
• Signal Processing

Background:

• Network interpretability is crucial for multi-modal image processing due to complex cross-modal dependencies.
• Existing multi-modal dictionary learning models are limited by single-layer and single-scale architectures, restricting their representational power.

Purpose of the Study:

• To introduce a multi-scale, multi-modal convolutional dictionary learning (M²CDL) model for enhanced representation in image processing.
• To propose a unified Deep M²CDL framework for multi-modal image restoration (MIR) and multi-modal image fusion (MIF) tasks.
• To ensure network interpretability by aligning the Deep M²CDL architecture with its optimization steps.

Main Methods:

• Developed a multi-layer M²CDL model for coarse-to-fine association of different image modalities.
• Created a unified Deep M²CDL framework by unfolding the M²CDL model, ensuring interpretable network modules.
• Learned dictionary and sparse feature priors directly through the network, avoiding handcrafted priors.

Main Results:

• The Deep M²CDL model demonstrated superior performance on various MIR and MIF tasks compared to state-of-the-art methods.
• Quantitative and qualitative evaluations confirmed the effectiveness of the proposed model.
• Visualizations of learned multi-modal sparse features and dictionary filters validated the network's interpretability.

Conclusions:

• The proposed Deep M²CDL framework offers an interpretable and effective solution for multi-modal image processing tasks.
• The multi-layer, multi-scale approach significantly improves representation ability.
• Learned priors contribute to better performance and interpretability in MIR and MIF.