Robust Neural Depth Prediction from Uncalibrated Small Motion Clip

Small camera movements often occur when taking shots with a handheld device due to unintentional hand shaking. While the parallax induced by the small motion provides a convenient means of reconstructing 3D scenes, the micro-baseline exacerbates noise in pixel-wise correspondences during triangulating 3D points, posing challenges for state-of-the-art 3D reconstruction approaches. Reconstruction approaches specifically designed for small motion scenes typically assume that camera poses or intrinsic parameters are known in advance, and most tackle the problem through a two-stage optimization pipeline involving bundle adjustment followed by dense reconstruction. These assumptions, however, restrict both the reconstruction accuracy and the range of applicable scenarios. In this study, we introduce an accurate and robust neural depth prediction method from uncalibrated small motion clips, termed Neu-DfUSMC. The proposed approach jointly estimates camera intrinsic parameters, poses, and depth maps from scratch within a neural rendering framework that exploits micro-baseline parallax cues. To improve the recovery of camera parameters and depth maps and to mitigate the risk of converging to poor local minima during photometric neural depth prediction, we incorporate a neural geometric constraint derived from keypoint correspondences. Furthermore, we present a random local regularization strategy that enforces first- and second-order derivative smoothness constraints, requiring the predicted depth at each pixel to be consistent with the depths of two randomly selected collinear neighboring pixels. This strategy substantially reduces noise in the predicted depth maps with only a modest increase in computational cost. In addition, the framework can seamlessly integrate available LiDAR depth or monocular depth estimates. We verify the superior performance of the method through qualitative and quantitative experiments on a variety of datasets, demonstrating its accuracy and robustness as a 3D reconstruction method applicable to real-world uncalibrated small motion clips.