MMCTNet: multimodal cross-scale transformer network for hyperspectral and LiDAR/SAR image classification

Area of Science:

• Remote Sensing
• Computer Vision
• Artificial Intelligence

Background:

• Multisource feature fusion is crucial for complex scene classification.
• Hyperspectral images (HSI) and LiDAR/SAR data offer complementary information.
• Existing methods face challenges in effectively fusing these diverse data sources.

Purpose of the Study:

• To propose a novel multimodal cross-scale transformer network (MMCTNet) for complex scene classification.
• To effectively fuse HSI, LiDAR, and SAR data for improved classification performance.
• To enhance the modeling of intra-modal spatial dependencies and cross-modal semantic complementation.

Main Methods:

• Developed a multimodal cross-scale transformer network (MMCTNet).
• Incorporated a spatial self-attention (SSA) module for intra-modal feature enhancement.
• Utilized a multiscale adaptive fusion (MSAF) module for cross-modal semantic complementation.
• Employed a transformer encoder with cross-attention for global semantic interaction.

Main Results:

• MMCTNet demonstrated superior performance on four public datasets (MUUFL, Augsburg, Berlin, 2018Houston).
• Achieved higher overall accuracy (OA), average accuracy (AA), and Kappa coefficient compared to existing methods.
• The proposed modules effectively enhanced feature fusion and classification accuracy.

Conclusions:

• MMCTNet offers an effective solution for multisource feature fusion in complex scene classification.
• The network architecture successfully integrates HSI, LiDAR, and SAR data.
• The approach significantly advances the state-of-the-art in remote sensing scene classification.