Multimodal Remote Sensing Image Segmentation With Intuition-Inspired Hypergraph Modeling.

Area of Science:

• Computer Vision
• Remote Sensing
• Artificial Intelligence

Background:

• Multimodal remote sensing (RS) image segmentation faces challenges in modeling intra- and inter-modal relationships due to object diversity and modal gaps.
• Existing methods often focus on single RS modalities, limiting their effectiveness in complex environments with noisy data and poor discrimination.
• Human cognitive processes, specifically guiding perception and integrative cognition, offer a model for understanding multimodal semantics.

Purpose of the Study:

• To develop a novel semantic understanding framework for multimodal RS segmentation inspired by human intuition.
• To address the limitations of single-modality approaches by effectively modeling relationships across multiple RS data types.
• To improve pixel-level semantic assignment in complex urban scenes using diverse RS data.

Main Methods:

• Proposed an intuition-inspired hypergraph network (I²HN) leveraging hypergraphs for modeling high-order relationships.
• Introduced a hypergraph parser to mimic guiding perception, learning intra-modal object-wise relationships and generating robust mono-modal representations.
• Developed a hypergraph matcher to simulate integrative cognition, dynamically updating hypergraph structures for improved cross-modal feature fusion.

Main Results:

• The proposed I²HN significantly outperforms state-of-the-art models on two multimodal RS datasets.
• Achieved high accuracy with F1/mIoU scores of 91.4%/82.9% on the ISPRS Vaihingen dataset.
• Attained F1/mIoU scores of 92.1%/84.2% on the MSAW dataset, demonstrating superior performance.

Conclusions:

• The intuition-inspired hypergraph network (I²HN) provides an effective framework for multimodal RS image segmentation.
• The method successfully models complex intra- and inter-modal relationships, enhancing semantic understanding of urban scenes.
• The proposed approach offers a promising direction for advancing multimodal remote sensing applications.