View Synthesis of Endoscope Images by Monocular Depth Prediction and Gaussian Splatting

We present a novel method for synthesising new views of endoscopic images, utilising an innovative 3D shape representation technique known as Gaussian splatting. Structure from Motion (SfM) has traditionally been employed for this purpose to recover 3D shapes of feature points and poses from video sequences, but has only been effective in limited situations due to the featureless, shiny and deformable organ internal surfaces of endoscopic images.To address this challenge, we propose a hybrid method that combines a single depth map estimation using deep neural networks (DNNs) with a Gaussian splatting approach. Our method incorporates 4D Gaussian splatting, optimising the shape and colour Gaussians of the point cloud to align with the input image and synthesising new views by considering deformations. Experiments demonstrate that the proposed method produces less degraded results compared to Gaussian splatting using SfM, a conventional method, using real endoscopic images, and is useful in cases where a conventional method, e.g. SfM, cannot be used.