NeeCo: Image Synthesis of Novel Instrument States Based on Dynamic and Deformable 3-D Gaussian Reconstruction

Area of Science:

• Medical Imaging
• Computer Vision
• Surgical Robotics

Background:

• Data-driven computer vision methods for surgical automation require extensive labeled datasets.
• Existing methods struggle with data scarcity in surgical imaging.

Purpose of the Study:

• To address data scarcity in surgical imaging datasets using a novel dynamic Gaussian Splatting technique.
• To generate realistic, labeled synthetic surgical images for training machine learning models.

Main Methods:

• Developed a dynamic Gaussian model to represent dynamic surgical scenes and render surgical instruments.
• Implemented a dynamic training adjustment strategy for camera pose calibration.
• Automatically generated annotations for synthetic data.
• Created a new dataset with 14,000 frames of surgical tool and camera motion.

Main Results:

• Generated photo-realistic labeled image datasets with a peak Signal-to-Noise Ratio (PSNR) of 29.87.
• Models trained on synthetically generated images outperformed those trained with standard data augmentation by 10%.
• Achieved an overall improvement of nearly 15% in model performance for surgical tasks.

Conclusions:

• The proposed dynamic Gaussian Splatting technique effectively generates high-quality synthetic surgical data, addressing data scarcity.
• Training on this synthetic data significantly enhances the performance of computer vision models in surgical automation.
• This approach offers a promising solution for improving surgical tool tracking, detection, and localization.