Multi-label remote sensing classification with self-supervised gated multi-modal transformers.

Area of Science:

• Remote Sensing
• Machine Learning
• Computer Vision

Background:

• Transformers show promise in machine learning and remote sensing (RS).
• RS research faces challenges due to limited labeled data and diverse data modes from various platforms.
• Self-supervised learning (SSL) offers a potential solution for RS data challenges.

Purpose of the Study:

• To develop an efficient multi-modal data fusion scheme for remote sensing.
• To address limitations in current multi-modal data fusion approaches in RS.

Main Methods:

• Proposed a multi-modal fusion mechanism based on gated unit control (MGSViT).
• Pretrained a Vision Transformer (ViT) model on the BigEarthNet dataset using two common SSL algorithms.
• Developed intra-modal and inter-modal gated fusion units for feature learning combining multispectral (MS) and synthetic aperture radar (SAR) data.

Main Results:

• The proposed MGSViT method effectively combines different modal data for key feature extraction.
• After fine-tuning, the method demonstrated superior performance compared to state-of-the-art algorithms in downstream classification tasks.
• Experimental results validated the effectiveness of the proposed fusion mechanism.

Conclusions:

• The MGSViT approach offers a significant advancement in multi-modal data fusion for remote sensing.
• The study validates the effectiveness of combining MS and SAR data through gated fusion units for improved RS data analysis.
• This research paves the way for more robust and accurate remote sensing applications.