A novel no-sensors 3D model reconstruction from monocular video frames for a dynamic environment

Area of Science:

• Computer Vision
• 3D Reconstruction
• Multimedia

Background:

• Occlusion poses significant challenges in dynamic environments for applications like multimedia and computer vision.
• Existing dense 3D reconstruction methods struggle with accuracy due to missing depth, camera pose, and motion data.
• Sensor-less 3D reconstruction is crucial for realistic interaction applications.

Purpose of the Study:

• To develop a novel framework for full 3D model reconstruction in complex dynamic scenes, specifically addressing the occlusion problem.
• To enable accurate 3D reconstruction using only monocular RGB video input, suitable for video streaming.
• To generate smooth and precise 3D point-clouds of dynamic environments without relying on sensor data.

Main Methods:

• The framework employs unsupervised learning to predict scene depth, camera pose, and object motion from monocular RGB videos.
• A two-phase approach involves initial prediction followed by frame-wise point cloud fusion for 3D model generation.
• Utilizes cumulative information from a sequence of RGB video frames for enhanced reconstruction.

Main Results:

• The proposed framework successfully reconstructs 3D models from dynamic scenes, effectively mitigating occlusion issues.
• Evaluations using localization error, RMSE, and fitness against LiDAR ground truth demonstrate high accuracy.
• Comparison with state-of-the-art methods like MRE and Chamfer Distance shows superior performance.

Conclusions:

• The developed framework provides a powerful, sensor-less solution for 3D model reconstruction in dynamic environments.
• It overcomes limitations of traditional methods by accurately handling occlusion using monocular video data.
• The approach is a promising candidate for applications requiring real-time 3D scene understanding and reconstruction.