HyFormer: Hybrid Transformer and CNN for Pixel-Level Multispectral Image Land Cover Classification.

Area of Science:

• Earth and Space Sciences
• Computer Science

Background:

• Convolutional Neural Networks (CNNs) struggle with pixelwise remote sensing (RS) classification and representing spectral sequence information.
• Existing methods often fail to adequately process pixel-level spectral data in RS images.

Purpose of the Study:

• To propose HyFormer, a novel multispectral RS image classification framework leveraging Transformer architecture.
• To overcome the limitations of traditional CNNs in handling pixelwise spectral data and enhance feature expressiveness.

Main Methods:

• A hybrid framework combining fully connected layers (FC) and CNNs to process 1D spectral sequences into 3D matrices.
• Integrating multi-level CNN features with linearly transformed spectral information as input for a Transformer encoder.
• Utilizing adjacent encoder skip connections for enhanced information fusion and MLP Head for pixel classification.

Main Results:

• HyFormer achieved 95.37% overall accuracy in Changxing County and 95.4% in Nanxun District using Sentinel-2 data.
• The proposed HyFormer framework demonstrated superior performance compared to the standard Vision Transformer (ViT).

Conclusions:

• HyFormer effectively enhances feature expressiveness and global modeling capabilities for multispectral RS image classification.
• The proposed framework offers a significant improvement over existing Transformer-based approaches for RS data analysis.