Advancing Hyperspectral and Multispectral Image Fusion: An Information-Aware Transformer-Based Unfolding Network

Area of Science:

• Remote Sensing
• Computer Vision
• Signal Processing

Background:

• Hyperspectral image (HSI) processing requires fusing high-resolution multispectral images (HR-MSI) with low-resolution HSI (LR-HSI) to generate high-resolution HSI (HR-HSI).
• Convolutional neural network (CNN)-based deep unfolding methods show promise but struggle with limited receptive fields and restricted feature transmission between stages, hindering performance.
• Accurate long-range spatial feature extraction and effective information transfer are critical for advanced HSI fusion.

Purpose of the Study:

• To propose a novel and efficient information-aware transformer-based unfolding network (ITU-Net) for MSI-HSI fusion.
• To address the limitations of CNNs in capturing long-range dependencies and facilitating cross-stage information flow.
• To improve the accuracy and performance of high-resolution hyperspectral image generation.

Main Methods:

• Developed a customized transformer block for learning spatial and frequency domain representations with linear complexity.
• Introduced information transfer guided linearized attention (ITLA) for efficient spatial feature extraction and contextual information transmission.
• Integrated frequency domain learning within the feedforward network (FFN) to capture image token variations and reduce frequency gaps.

Main Results:

• The proposed ITU-Net effectively models long-range dependencies in hyperspectral data.
• ITLA enables high-throughput information transfer between adjacent stages, enhancing spatial feature extraction.
• Frequency domain learning successfully captures image variations and bridges frequency gaps, improving fusion quality.

Conclusions:

• ITU-Net achieves state-of-the-art (SOTA) performance on both synthetic and real hyperspectral datasets.
• The transformer-based approach overcomes CNN limitations in HSI fusion, particularly for long-range spatial features.
• The proposed methods offer a significant advancement in high-resolution hyperspectral image reconstruction.