ESMStereo: Enhanced ShuffleMixer Disparity Upsampling for Real-Time and Accurate Stereo Matching

Area of Science:

• Computer Vision
• Deep Learning
• Autonomous Systems

Background:

• Deep learning-based stereo matching is crucial for autonomous systems but faces challenges in achieving both high accuracy and real-time performance.
• Large-scale cost volumes improve accuracy but hinder real-time processing due to redundancy and computational intensity.
• Small-scale cost volumes enable real-time performance but often lack sufficient information for accurate disparity estimation.

Purpose of the Study:

• To address the trade-off between accuracy and real-time performance in stereo matching.
• To propose a novel deep learning module, the Enhanced Shuffle Mixer (ESM), to mitigate information loss in small-scale cost volumes.
• To improve disparity estimation accuracy while maintaining real-time capabilities in computer vision models.

Main Methods:

• Developed the Enhanced Shuffle Mixer (ESM) to integrate primary features into the disparity upsampling unit, restoring lost details.
• Implemented a feature fusion mechanism combining initial disparity estimation with image features.
• Utilized shuffling, layer splitting, and a compact feature-guided hourglass network for refining scene geometry.
• Focused on local contextual connectivity with a large receptive field and low computational cost.

Main Results:

• The proposed ESM effectively mitigates information loss associated with small-scale cost volumes.
• ESM enhances disparity estimation accuracy by recovering detailed scene geometry.
• The compact ESMStereo model achieved high inference speeds: 116 FPS on RTX 4070S and 91 FPS on AGX Orin.
• Demonstrated improved accuracy while maintaining real-time performance.

Conclusions:

• The Enhanced Shuffle Mixer (ESM) offers a viable solution for real-time, high-accuracy stereo matching in computer vision.
• ESM's architecture successfully balances the need for detailed feature representation with computational efficiency.
• The approach shows significant promise for advancing the capabilities of modern autonomous systems.