Deep Semi-Smooth Newton-Driven Unfolding Network for Multi-Modal Image Super-Resolution.

Area of Science:

• Computer Vision
• Machine Learning
• Image Processing

Background:

• Deep unfolding methods enhance Multi-modal Image Super-Resolution (MISR) by integrating cross-modal priors.
• Existing deep unfolding approaches use first-order optimization, limiting learning efficiency and reconstruction accuracy.

Purpose of the Study:

• To propose a novel Semi-smooth Newton driven Unfolding network for MISR (SNUM-Net).
• To overcome the limitations of first-order optimization in deep unfolding MISR methods.

Main Methods:

• Developed a Semi-smooth Newton-driven MISR (SNM) algorithm as a theoretical foundation.
• Unfolded the iterative solution of the SNM algorithm into a novel deep network (SNUM-Net).
• SNUM-Net is the first deep unfolding MISR network based on second-order optimization.

Main Results:

• SNUM-Net offers a universal paradigm for diverse MISR tasks without scenario-specific constraints.
• The network provides an explainable framework with clear mathematical correspondence to the SNM algorithm.
• Evaluations on 10 datasets across 3 MISR tasks show superior reconstruction accuracy and generalization capability.

Conclusions:

• SNUM-Net represents a significant advancement in deep unfolding for MISR.
• The proposed second-order optimization approach enhances performance and explainability.
• SNUM-Net demonstrates strong potential for various MISR applications.