Joint Classification of Hyperspectral Images and LiDAR Data Based on Dual-Branch Transformer

Area of Science:

• Remote Sensing
• Geospatial Analysis
• Machine Learning

Background:

• Single-modality data limits classification performance in complex scenarios.
• Multimodal remote sensing data integration faces challenges like sample differences and feature correlation.
• Effective interaction between heterogeneous data sources is crucial for enhanced classification.

Purpose of the Study:

• To propose a novel dual-branch cross-Transformer feature fusion network for joint land cover classification.
• To effectively integrate hyperspectral imagery (HSI) and Light Detection and Ranging (LiDAR) data.
• To overcome limitations of single-modality approaches and enhance classification accuracy.

Main Methods:

• A dual-branch network leveraging convolutional operators for spatial features and Transformer architecture for remote dependencies.
• An improved self-attention mechanism for intra-modality feature aggregation (HSI spectral, LiDAR elevation).
• A cross-attention based feature fusion module for inter-modality information integration.

Main Results:

• The proposed network effectively fuses spectral and spatial information from HSI and LiDAR data.
• Experimental results on three datasets demonstrate superior performance compared to existing methods.
• The cross-attention mechanism facilitates complementary information exchange between modalities.

Conclusions:

• The dual-branch cross-Transformer network offers a powerful approach for multimodal remote sensing classification.
• Jointly utilizing HSI and LiDAR data with effective fusion enhances land cover classification accuracy.
• The model addresses key challenges in multimodal data integration for surface observation.