Z-Splat: Z-Axis Gaussian Splatting for Camera-Sonar Fusion.

Area of Science:

• Computer Vision
• 3D Graphics
• Robotics

Background:

• Differentiable 3D-Gaussian Splatting (GS) is a key technique for 3D scene reconstruction from images.
• Restricted imaging scenarios (e.g., underwater, indoor) cause the 'missing cone' problem in GS, degrading depth reconstruction.
• Existing GS methods struggle with limited viewpoints, leading to incomplete 3D scene data.

Purpose of the Study:

• To address the 'missing cone' problem in 3D-Gaussian Splatting using complementary sensor data.
• To improve 3D scene reconstruction accuracy and novel view synthesis in challenging imaging environments.
• To develop fusion algorithms combining RGB camera and sonar data for enhanced 3D scene representation.

Main Methods:

• Extended Gaussian Splatting algorithms to incorporate data from two common sonar types.
• Developed novel fusion algorithms to simultaneously process RGB camera and sonar data.
• Validated the approach through simulations, emulations, and hardware experiments in diverse scenarios.

Main Results:

• Achieved a 5 dB improvement in Peak Signal-to-Noise Ratio (PSNR) for novel view synthesis.
• Reduced Chamfer distance by 60% for 3D geometry reconstruction.
• Demonstrated superior performance of the fusion method over traditional GS in restricted baseline scenarios.

Conclusions:

• Integrating sonar transient data effectively resolves the 'missing cone' issue in 3D-Gaussian Splatting.
• The proposed fusion approach significantly enhances both the quality of synthesized novel views and the accuracy of 3D geometry.
• This method offers a robust solution for 3D scene reconstruction in real-world applications with limited camera viewpoints.