Rethinking Multi-modal Image Super-resolution: The Key Role of Cross-modal Consistency Prior

Area of Science:

• Computer Vision
• Image Processing

Background:

• Multi-modal image super-resolution (MISR) relies on cross-modal consistency.
• Existing methods often fail to preserve high-frequency details and generalize well, leading to image artifacts.

Purpose of the Study:

• To investigate the distribution of the modality gap in Laplacian responses for MISR.
• To propose a novel consistency prior that improves super-resolution performance.
• To develop an interpretable deep learning model for MISR.

Main Methods:

• Proposed a T-distribution formed Laplacian response Consistency (TLC) model.
• Introduced a T-distribution based Multi-modal Consistency (TMC) prior.
• Incorporated a Multiplicative Degradation (MD) matrix for adaptive degradation modeling.
• Unfolded the TLC model into an interpretable network, TLCNet.

Main Results:

• Demonstrated that the modality gap in Laplacian responses follows a T-distribution, not Gaussian or Laplacian.
• TLCNet achieved superior super-resolution performance across nine datasets and three MISR tasks.
• Showcased improved preservation of high-frequency components and texture accuracy.

Conclusions:

• The T-distribution is a more suitable prior for modeling the modality gap in MISR.
• The proposed TLC model and TLCNet offer a significant advancement in MISR.
• The model's interpretability aids in understanding guidance image influence.