High-Quality Novel View Synthesis of Robotic Surgical Scenes using Gaussian Splatting with Depth Prior

Robot-assisted surgery has revolutionized modern medical procedures by enhancing precision, reducing invasiveness, and providing a clearer, more controlled environment. However, it still faces challenges in fully visualizing the target tissue, particularly from multiple perspectives. This limitation is most evident in minimally invasive surgeries, Therefore, the ability to synthesize new views of the surgical scene is becoming increasingly critical. By generating multi-view visualizations, surgeons can gain a more comprehensive understanding of the target tissue, improving spatial awareness and decision-making during surgery.

This article proposes an innovative novel view synthesis method for robotic surgical scenarios, which utilizes pre-trained depth estimation model to obtain global depth information and solves the scale ambiguity problem encountered in the transition region of the Gaussian distribution in the 3D Gaussian model. In addition, we introduce a multi-scale loss optimization strategy that captures features of various scales through a multi-scale loss function to regularize the Gaussian parameters while maintaining the 3D consistency of the splatting.

Our method is evaluated against current scene novel view synthesis techniques using our robotic surgery scene dataset, along with the Hamlyn and Stereo MIS public datasets. The proposed approach achieved an average PSNR of 33.45, SSIM of 0.939, LPIPS of 0.153, and an RMSE of 0.022 across the three datasets.

Our approach helps to enhance the visualization capabilities of robotic surgical systems by synthesizing novel views of surgical scenes. A deeper understanding of the target tissue will enhance patient safety during surgery and support surgeon training. These advancements will contribute to the improvement of robot-assisted surgery, making it more adaptable to diverse clinical scenarios.