Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal...

Area of Science:

• Computer Vision
• Geometric Deep Learning
• 3D Reconstruction

Background:

• Monocular depth estimation methods often generalize poorly to real-world metric scales.
• Surface normal estimation faces challenges in zero-shot performance due to limited labeled data.
• Accurate 3D recovery from single images requires precise metric depth and surface normal information.

Purpose of the Study:

• To introduce Metric3D v2, a geometric foundation model for zero-shot metric depth and surface normal estimation.
• To address the limitations of existing methods in recovering metric scale and achieving robust zero-shot generalization.
• To enable accurate single-image metrology and improve monocular Simultaneous Localization and Mapping (SLAM).

Main Methods:

• Developed a canonical camera space transformation module to resolve metric ambiguity in depth estimation.
• Introduced a joint depth-normal optimization module leveraging metric depth data to enhance normal estimation.
• Trained the model on over 16 million images from diverse camera models with varied annotations.

Main Results:

• Metric3D v2 achieves state-of-the-art zero-shot generalization to new camera settings.
• The model ranks first in multiple zero-shot and standard benchmarks for metric depth and surface normal prediction.
• Demonstrated accurate metric 3D structure recovery from internet images and reduced scale drift in monocular SLAM.

Conclusions:

• Metric3D v2 serves as a versatile geometric foundation model for single-image 3D recovery.
• The proposed methods significantly improve metric depth and surface normal estimation accuracy in a zero-shot setting.
• The model paves the way for advanced applications in single-image metrology and dense 3D mapping.