AIM-SEEM: Adapting SEEM for Open-Vocabulary Terrain Segmentation Across Arbitrary Imaging Modalities

Area of Science:

• Robotics
• Computer Vision
• Artificial Intelligence

Background:

• Terrain segmentation is crucial for robotic environmental perception and decision-making.
• Existing methods fail in real-world outdoor scenarios due to fixed sensing configurations and limited semantic classes.

Purpose of the Study:

• To systematically study open-vocabulary terrain segmentation under arbitrary imaging modality combinations.
• To propose a unified foundation model-based framework, AIM-SEEM, for robust terrain segmentation.

Main Methods:

• Developed AIM-SEEM (SEEM for Arbitrary Imaging Modalities) based on the Segment Everything Everywhere All at Once (SEEM) model.
• Implemented stable input side adaptation and controlled fusion of heterogeneous modalities.
• Introduced a vision-guided text calibration mechanism to address distribution shifts and preserve open-vocabulary segmentation.

Main Results:

• AIM-SEEM demonstrated stable adaptation to various modality combinations and counts.
• The vision-guided text calibration mechanism effectively preserved open-vocabulary segmentation under multi-modality inputs.
• Experiments showed consistent outperformance of AIM-SEEM over prior methods in full-modality, modality-agnostic, and open-vocabulary settings.

Conclusions:

• AIM-SEEM provides a unified foundation model for open-vocabulary terrain segmentation adaptable to diverse and dynamic outdoor environments.
• The framework effectively leverages pre-trained visual priors and addresses challenges posed by modality extension and distribution shifts.
• AIM-SEEM significantly advances the reliability and flexibility of robotic environmental perception systems.