4D-aware stereo matching via implicit spectral reconstruction with multi-modal training and RGB-only deployment

Area of Science:

• Computer Vision
• Computational Imaging
• 3D Reconstruction

Background:

• Conventional RGB stereo matching faces challenges with metamerism and weak textures, limiting depth estimation accuracy.
• Implicit spectral information recovery offers a potential solution without requiring specialized spectral sensors.

Purpose of the Study:

• To enhance depth estimation in stereo matching by leveraging implicitly recovered spectral information.
• To introduce a novel reconstruction-to-matching architecture and a data augmentation strategy for improved geometric perception.

Main Methods:

• A reconstruction-to-matching architecture was developed to map RGB inputs into a latent spectral space, incorporating material priors.
• A co-aperture system combining a liquid crystal tunable filter and Scheimpflug LiDAR was used for pixel-aligned multimodal acquisition.
• A "Measure-and-Complete" strategy was proposed to generate dense pseudo-ground truth data from sparse LiDAR scans.

Main Results:

• The proposed method demonstrated a 4.34% reduction in endpoint error compared to conventional methods.
• The integration of spectral priors significantly improved the accuracy of depth estimation.
• The "Measure-and-Complete" strategy effectively addressed data scarcity for training.

Conclusions:

• Implicitly recovered spectral information can significantly enhance depth estimation in stereo matching.
• The developed architecture and data acquisition/generation methods are effective for improving geometric perception.
• This approach offers a practical way to improve 3D reconstruction using readily available RGB data augmented with spectral insights.