A spatial-spectral fusion convolutional transformer network with contextual multi-head self-attention for...

Area of Science:

• Remote Sensing
• Computer Vision
• Artificial Intelligence

Background:

• Hyperspectral image (HSI) classification benefits from combining Convolutional Neural Networks (CNNs) for local features and Vision Transformers for global features.
• Existing fusion methods often introduce inductive biases and do not sufficiently enhance Transformer's local contextual information extraction via convolutional embedding.

Purpose of the Study:

• To propose a novel spatial-spectral fusion convolutional Transformer (SSFCT) method with contextual multi-head self-attention (CMHSA) for improved HSI classification.
• To enhance the extraction of local contextual information within Transformer architectures for HSI analysis.

Main Methods:

• A local feature aggregation (LFA) module using a three-branch convolution and attention layers to extract and refine local spatial-spectral features.
• A contextual multi-head self-attention (CMHSA) mechanism integrating static and dynamic local contextual representations from 3D convolution and attention.
• A dual-branch spatial-spectral convolutional Transformer (DSSCT) module and an attention feature fusion (AFF) module for comprehensive global-local feature extraction.

Main Results:

• The proposed SSFCT method achieved state-of-the-art performance on five diverse HSI datasets.
• Achieved high overall accuracies: 98.03% (Indian Pines), 99.68% (Salinas Valley), 98.65% (Houston2013), 97.97% (Botswana), and 89.43% (Yellow River Delta).

Conclusions:

• The SSFCT method effectively captures global-local associations in both spatial and spectral domains for HSI classification.
• The integration of enhanced local feature extraction and Transformer capabilities significantly improves classification accuracy.