RayMVSNet++: Learning Ray-Based 1D Implicit Fields for Accurate Multi-View Stereo.

Area of Science:

• Computer Vision
• Machine Learning
• 3D Reconstruction

Background:

• Learning-based multi-view stereo (MVS) methods often rely on computationally intensive 3D convolutions.
• High computational and memory demands limit the resolution of depth maps in existing MVS techniques.
• Optimizing cost volumes directly is resource-prohibitive for high-resolution outputs.

Purpose of the Study:

• To develop a more efficient and accurate learning-based MVS method.
• To reduce the computational complexity associated with traditional MVS approaches.
• To enable high-quality depth estimation and 3D reconstruction in challenging scenarios.

Main Methods:

• Proposes RayMVSNet, which optimizes depth along camera rays by learning a 1D implicit field.
• Utilizes transformer features for sequential ray-based depth prediction, mimicking epipolar line search.
• Incorporates multi-task learning and leverages the monotonicity of Signed Distance Functions (SDFs) for improved accuracy.
• Introduces RayMVSNet++ with an attentional gating unit for enhanced contextual feature aggregation.

Main Results:

• RayMVSNet achieves top rankings on DTU and Tanks & Temples datasets, with scores of 0.33 mm and 59.48% F-score, respectively.
• Demonstrates high-quality depth estimation and point cloud reconstruction for non-textured, occluded, and varying depth scenes.
• RayMVSNet++ attains state-of-the-art performance on ScanNet, achieving an AbsRel of 0.058m.
• Achieves accurate results on textureless regions and scenes with large depth variations.

Conclusions:

• Ray-based depth optimization offers a lightweight and effective alternative to cost volume optimization in MVS.
• The proposed RayMVSNet and RayMVSNet++ significantly advance the state-of-the-art in learning-based MVS.
• The methods are robust to challenging real-world conditions, including poor lighting and motion blur.