Sparse Self-Prompt-Guided Stereo Matching for Real-World Generalization.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Stereo matching models excel on benchmarks but struggle in real-world, unconstrained environments.
• Deploying stereo matching models in diverse conditions requires robust generalization capabilities.

Purpose of the Study:

• To present a novel sparse self-prompt-guided network (SSPGNet) for stereo matching.
• To improve the generalization of stereo matching models across diverse indoor and outdoor environments.
• To enable direct deployment of stereo matching in real-world perception systems.

Main Methods:

• Introduced a sparse self-prompt guidance mechanism using a self-estimated sparse disparity map from visual foundation model features.
• Employed a sparse-to-dense prediction approach, refining sparse disparity into dense maps via cross-attention stereo feature interaction.
• Collected a diverse dataset of indoor and outdoor stereo pairs using a ZED 2 camera for real-world evaluation.

Main Results:

• SSPGNet demonstrated strong performance on public benchmarks (KITTI, Middlebury, ETH3D) and the in-the-wild dataset.
• Achieved top rankings on three out of four public benchmarks under a cross-domain (zero-shot) protocol.
• Showcased preservation of semantic awareness from visual foundation models and enhanced stereo correspondence reasoning.

Conclusions:

• The proposed sparse-to-dense prompt mechanism significantly enhances stereo matching performance and generalization.
• SSPGNet shows great potential for real-world stereo perception system deployment.
• The model's ability to generalize across diverse environments addresses a key challenge in stereo matching.