Multimodal Deep Unfolding for Guided Image Super-Resolution.

Area of Science:

• Computer Vision
• Machine Learning
• Image Processing

Background:

• Image super-resolution is an ill-posed inverse problem.
• Deep learning models typically learn end-to-end mappings from low-resolution to high-resolution images.
• Existing multimodal deep learning models often lack domain-specific knowledge integration.

Purpose of the Study:

• To propose a novel multimodal deep learning design for image super-resolution.
• To incorporate domain knowledge, specifically sparse priors, into the network architecture.
• To effectively integrate information from an additional image modality (side information) for improved reconstruction.

Main Methods:

• A novel deep unfolding operator inspired by iterative algorithms for convolutional sparse coding with side information was developed.
• The deep unfolding architecture was integrated into a multimodal framework for guided image super-resolution.
• Residual learning was explored as an alternative multimodal design for enhanced training efficiency.

Main Results:

• The proposed interpretable deep unfolding network demonstrated superior performance in guided image super-resolution.
• The multimodal approach achieved state-of-the-art results in super-resolution for near-infrared and multi-spectral images.
• Effective depth upsampling was demonstrated using RGB images as side information.

Conclusions:

• The developed multimodal deep learning framework with interpretable deep unfolding significantly advances image super-resolution.
• Incorporating domain knowledge and leveraging multi-modal information leads to superior reconstruction quality.
• The proposed method offers a powerful and versatile solution for various image enhancement tasks.