REST: Holistic Learning for End-to-End Semantic Segmentation of Whole-Scene Remote Sensing Imagery.

Area of Science:

• Computer Vision
• Remote Sensing
• Deep Learning

Background:

• Semantic segmentation of remote sensing imagery (RSI) is crucial for pixel-level classification.
• Large whole-scene RSI (WRI) presents memory challenges for conventional deep learning models, leading to suboptimal cropping or fusion strategies.
• Existing methods often degrade performance when handling large-scale WRI due to GPU memory constraints.

Purpose of the Study:

• Introduce Robust End-to-end semantic Segmentation (REST), the first intrinsically end-to-end framework for holistic WRI segmentation.
• Overcome GPU memory limitations to enable efficient and effective processing of WRI.
• Provide a versatile and robust solution for diverse WRI segmentation tasks.

Main Methods:

• Developed the Robust End-to-end semantic Segmentation (REST) architecture for holistic WRI segmentation.
• Proposed a novel spatial parallel interaction mechanism (SPIM) to address GPU memory constraints and achieve global context awareness.
• Designed REST as a plug-and-play framework compatible with various encoders, decoders, and foundation models.

Main Results:

• REST enables truly holistic segmentation of WRI, outperforming cropping-based and fusion-based methods.
• The spatial parallel interaction mechanism (SPIM) effectively overcomes GPU memory limitations.
• REST demonstrates near-linear throughput scalability with additional GPUs and achieves consistent performance across various scenarios (single/multi-class, multispectral/hyperspectral, satellite/drone).

Conclusions:

• REST offers a robust and versatile solution for the holistic segmentation of whole-scene remote sensing imagery.
• The framework's plug-and-play nature and efficient processing mechanism pave the way for advanced WRI analysis.
• REST shows potential for extension to other large-size imagery segmentation tasks, such as medical imaging.