Gaussian Splatting with a Shading Model for Endoscope Image Visualization

Area of Science:

• Computer Vision
• Computer Graphics
• Medical Imaging

Background:

• Gaussian splatting is a 3D representation technique using optimized Gaussian clouds for novel view synthesis.
• Static illumination is a key assumption in Gaussian splatting, which is violated in endoscopy due to the camera-attached light source.
• This violation leads to visual artifacts, specifically 'floating artifact Gaussians', degrading reconstruction quality.

Purpose of the Study:

• To address the limitations of Gaussian splatting in dynamic illumination scenarios, particularly in endoscopy.
• To develop a rendering method that accurately represents 3D scenes under varying lighting conditions.
• To mitigate the 'floating artifact Gaussians' problem in endoscopic 3D reconstruction.

Main Methods:

• A novel shader was developed and applied to the output of a Gaussian renderer.
• The shader incorporates physically-based rendering components: diffuse and specular reflection models, and vignetting.
• The shader and Gaussian model were co-optimized against ground truth images for accurate reconstruction.

Main Results:

• The proposed shader-based method effectively reduces artifacts caused by dynamic illumination in Gaussian splatting.
• The technique demonstrated successful application on both synthetic and real-world endoscopic image datasets.
• Visual quality and geometric accuracy of the 3D reconstructions were significantly improved.

Conclusions:

• The shader integration offers a robust solution for enhancing Gaussian splatting in challenging endoscopic imaging environments.
• This approach improves the fidelity of 3D models generated from endoscopic data.
• The method shows promise for various applications requiring accurate 3D reconstruction from images with dynamic lighting.