Arbitrary Optics for Gaussian Splatting Using Space Warping

Area of Science:

• Computer Vision
• Computer Graphics
• 3D Reconstruction

Background:

• Recent advances enable rapid, real-time 3D scene reconstruction from RGB images.
• 3D Gaussian splatting offers superior performance and reduced computational cost compared to prior methods.
• Current 3D Gaussian splatting relies on a restrictive pinhole camera model.

Purpose of the Study:

• To extend 3D Gaussian splatting to support arbitrary camera optics, including fisheye lenses.
• To improve the accuracy and robustness of 3D scene reconstruction from highly distorted images.
• To reduce artifacts in outdoor scene reconstructions.

Main Methods:

• Introduced a differentiable warping function to adapt the Gaussian scene representation for non-pinhole cameras.
• Implemented a learnable skybox to mitigate overfitting and floating artifacts in outdoor environments.
• Validated the approach using both synthetic and real-world image datasets.

Main Results:

• Successfully reconstructed accurate 3D scenes from images captured with highly distorting camera lenses.
• Demonstrated the ability to render photorealistic images from these non-standard reconstructions.
• Showcased reduced artifacts in outdoor scene reconstructions due to the learnable skybox.

Conclusions:

• The proposed extension effectively removes the pinhole camera constraint in 3D Gaussian splatting.
• The method enables high-fidelity 3D reconstruction and rendering with arbitrary camera models.
• This work broadens the applicability of 3D Gaussian splatting for diverse real-world scenarios.