Fractional Fourier Image Transformer for Multimodal Remote Sensing Data Classification.

Area of Science:

• Remote Sensing
• Computer Vision
• Deep Learning

Background:

• Deep learning excels at hyperspectral image (HSI) and light detection and ranging (LiDAR) data classification by extracting local semantic features.
• Convolutional Neural Networks (CNNs) have limited receptive fields for global context, while Vision Transformers (ViTs) struggle with local semantic information.

Purpose of the Study:

• To propose a novel backbone network, the fractional Fourier image transformer (FrIT), for effective extraction of both global and local contexts in HSI and LiDAR data.
• To address the limitations of existing deep learning models in capturing comprehensive contextual information for fused HSI and LiDAR data classification.

Main Methods:

• Pixel-level fusion of HSI and LiDAR data.
• Utilizing multisource and HSI feature extractors for local context capture.
• Employing a plug-and-play fractional Fourier image transformer (FrIT) for global contextual and sequential feature extraction.
• Connecting contextual features across multiple fractional domains to minimize information loss.

Main Results:

• The proposed FrIT framework significantly enhances the extraction of both local and global contextual information.
• FrIT demonstrates improved performance in HSI and LiDAR data classification compared to traditional CNNs and ViTs.
• Experiments on five HSI and LiDAR scenes, including a new benchmark, validate the effectiveness of FrIT.

Conclusions:

• The fractional Fourier image transformer (FrIT) offers an effective and efficient solution for joint HSI and LiDAR data classification.
• FrIT overcomes the limitations of CNNs and ViTs by integrating local and global feature extraction capabilities.
• The proposed method shows promising results and potential for advancing remote sensing data analysis.