Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Multi-modal image restoration (MIR) and fusion (MIF) are challenging tasks requiring effective information integration from diverse image sources.
• Existing deep learning methods often struggle to optimally leverage shared and modality-specific information.

Purpose of the Study:

• To propose a novel deep convolutional neural network, CU-Net, for general MIR and MIF tasks.
• To design a network architecture inspired by multi-modal convolutional sparse coding (MCSC) for improved performance.

Main Methods:

• Developed CU-Net (common and unique information splitting network) based on the MCSC model.
• CU-Net comprises three modules: unique feature extraction (UFEM), common feature preservation (CFPM), and image reconstruction (IRM).
• Each module incorporates learned convolutional sparse coding (LCSC) blocks.

Main Results:

• Demonstrated the effectiveness of CU-Net on various MIR and MIF tasks through extensive numerical results.
• Achieved state-of-the-art performance in tasks such as RGB-guided depth super-resolution and flash-guided denoising.
• Validated successful application in multi-focus and multi-exposure image fusion.

Conclusions:

• CU-Net offers a powerful new approach for multi-modal image restoration and fusion.
• The network's ability to automatically split common and unique information is key to its success.
• The proposed method shows significant potential for advancing image processing technologies.