3D Reconstruction Using 3D Registration-Based ToF-Stereo Fusion

Area of Science:

• Computer Vision
• 3D Sensing Technologies
• Robotics and Perception

Background:

• Accurate depth sensing is crucial for immersive technologies like Augmented Reality (AR), Extended Reality (XR), and the Metaverse.
• Traditional depth acquisition methods using stereo cameras or Time-of-Flight (ToF) sensors have limitations in accuracy and completeness.
• Integrating multiple depth sensing modalities offers a promising approach to overcome individual sensor drawbacks.

Purpose of the Study:

• To develop and validate a novel sensor fusion technique for enhanced 3D reconstruction accuracy.
• To improve the quality of depth maps by combining stereo vision and ToF sensing.
• To generate high-fidelity 3D mesh data from fused sensor inputs.

Main Methods:

• A generalized multi-camera calibration method incorporating both color and depth data was employed.
• Depth maps from stereo and ToF sensors were fused using 3D registration and reprojection techniques.
• Hole-filling algorithms refined the fused depth map, followed by surface reconstruction for mesh generation.

Main Results:

• The proposed ToF-stereo fusion method significantly improved depth map accuracy compared to individual sensors.
• Hole-filling effectively addressed data gaps in the fused depth map.
• The final mesh data demonstrated high fidelity and accuracy in real-world tests.

Conclusions:

• Complementary fusion of stereo and ToF depth data provides superior 3D information accuracy.
• The developed pipeline offers a robust solution for high-quality 3D reconstruction in AR/XR applications.
• The method's efficiency was validated through comparative analysis with existing algorithms.