Lightweight and Accurate Multi-View Stereo With Confidence-Aware Diffusion Model.

Area of Science:

• Computer Vision
• 3D Reconstruction
• Machine Learning

Background:

• Multi-view stereo (MVS) methods reconstruct 3D geometry from calibrated images, often by estimating and fusing depth maps.
• Improving computational efficiency in MVS is crucial, with common approaches involving coarse-to-fine depth map refinement.
• Diffusion models have shown remarkable success in generative tasks through iterative denoising.

Purpose of the Study:

• To propose a novel MVS framework integrating diffusion models for enhanced 3D reconstruction.
• To formulate depth refinement as a conditional diffusion process guided by a specialized condition encoder.
• To improve efficiency and performance in MVS through innovative network architectures and sampling strategies.

Main Methods:

• Introduced a conditional diffusion process for depth refinement in MVS.
• Designed a condition encoder to guide the diffusion process, leveraging depth estimation characteristics.
• Developed an efficient diffusion network combining lightweight 2D U-Net and convolutional GRU.
• Proposed a confidence-based sampling strategy for adaptive depth hypothesis selection.

Main Results:

• The proposed framework led to two new MVS methods: DiffMVS and CasDiffMVS.
• DiffMVS demonstrated competitive performance with state-of-the-art efficiency in runtime and GPU memory usage.
• CasDiffMVS achieved state-of-the-art results on benchmark datasets including DTU, Tanks & Temples, and ETH3D.

Conclusions:

• The integration of diffusion models offers a promising direction for advancing MVS techniques.
• The developed methods provide efficient and high-performance solutions for 3D geometry reconstruction.
• The proposed framework and strategies pave the way for future research in diffusion-based MVS.