T-Rex2++: Toward Generic Object Perception via Text-Visual Prompt Synergy.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Traditional object perception methods struggle with rare objects using text prompts or novel objects using visual prompts.
• Existing approaches lack a unified framework for both abstract concept representation and concrete visual depiction.

Purpose of the Study:

• To introduce T-Rex2++, a unified framework for open-set object detection and instance segmentation.
• To develop a text-visual synergy mechanism for enhanced object representation.
• To enable autonomous object discovery and localization without user cues.

Main Methods:

• Implemented a text-visual synergy mechanism using contrastive learning to align modalities in a shared feature space.
• Introduced a learnable Universal Prompt to model generic objectness for autonomous perception.
• Integrated a zero-shot instance segmentation module for pixel-level synergy verification.

Main Results:

• T-Rex2++ demonstrates robust zero-shot object perception across diverse scenarios.
• The text-visual synergy effectively represents both abstract and concrete object properties.
• The Universal Prompt enables autonomous discovery and localization of arbitrary objects.

Conclusions:

• T-Rex2++ provides a versatile foundation for generic open-set object perception.
• The proposed framework successfully integrates text and visual modalities for superior performance.
• Autonomous perception capabilities are significantly advanced by the Universal Prompt.