MMFormer: Multi-Modality semi-Supervised vision transformer in remote sensing imagery classification.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Transformer architectures are advancing multimodal tasks.
• Current fusion methods neglect feature complementarity and consistency, causing redundant fusion and incomplete representations.

Purpose of the Study:

• Introduce MMFormer, a novel semi-supervised algorithm for high-dimensional multimodal fusion.
• Enhance interactivity between modal mappings for comprehensive representations.
• Address challenges of sparse annotations in high-dimensional multimodal data.

Main Methods:

• Inspired by topological homology groups, MMFormer uses a complete dictionary lookup and homology space for representational consistency.
• Employs an exclusivity-aware mapping to highlight complementary modal information.
• Introduces a consistency joint regularization term to mitigate sparse annotation issues.

Main Results:

• MMFormer demonstrates superiority across three benchmarks (Houston2013, Augsburg, MUUFL).
• Achieved accuracy improvements of 3.12% (Houston2013), 1.86% (Augsburg), and 1.66% (MUUFL).
• Confirms robustness and effectiveness under sparse annotation conditions.

Conclusions:

• MMFormer offers a unified end-to-end optimization framework for high-dimensional multimodal fusion.
• Represents the first application of semi-supervised visual transformers in this domain.
• Effectively enhances multimodal feature interpretation and fusion accuracy.