SSTF-Unet: Spatial-Spectral Transformer-Based U-Net for High-Resolution Hyperspectral Image Acquisition

Area of Science:

• Remote Sensing
• Computer Vision
• Image Processing

Background:

• Hyperspectral image (HSI) and multispectral image (MSI) fusion is crucial for generating high-resolution HSI (HR-HSI).
• Existing Convolutional Neural Network (CNN)-based fusion methods struggle to capture global features due to kernel limitations.

Purpose of the Study:

• To develop an advanced fusion technique that overcomes the limitations of CNNs in capturing global spatial-spectral information.
• To improve the accuracy and performance of HR-HSI generation through effective image fusion.

Main Methods:

• A novel Spatial-Spectral Transformer-based U-Net (SSTF-Unet) architecture is proposed.
• The SSTF-Unet incorporates Spatial Transformer Blocks (SATB) and Spectral Transformer Blocks (SETB) for parallel spatial and spectral self-attention.
• Multiple Spatial-Spectral Fusion Blocks (SSFB) are stacked in a U-Net structure for multiscale feature fusion.

Main Results:

• The SSTF-Unet effectively captures long-range dependencies and intrinsic image information.
• Experimental results on public HSI datasets show superior performance compared to existing HSI and MSI fusion methods.
• The proposed method demonstrates enhanced accuracy in HR-HSI generation.

Conclusions:

• The SSTF-Unet provides a significant advancement in hyperspectral and multispectral image fusion.
• This transformer-based approach offers a more robust solution for generating high-resolution hyperspectral images.
• The method's ability to capture global features leads to improved fusion outcomes.