Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding

Area of Science:

• Robotics
• Computer Vision
• Artificial Intelligence

Background:

• Surgical scene understanding is crucial for situation-aware robotic surgeries and training.
• Domain shifts and new instruments pose challenges for current robotic surgery models.
• Domain generalization (DG) is vital for expanding instrument-tissue interaction detection to new surgical domains.

Purpose of the Study:

• To develop an incremental domain generalization (DG) approach for predicting instrument-tissue interactions in robot-assisted surgery.
• To enable robotic systems to learn new surgical skills incrementally without forgetting existing ones.
• To improve the adaptability and performance of robotic surgery systems in diverse surgical environments.

Main Methods:

• Employed incremental DG on scene graphs, incorporating incremental learning (IL) and knowledge distillation.
• Designed an enhanced curriculum by smoothing (E-CBS) using Laplacian of Gaussian (LoG) and Gaussian kernels.
• Integrated E-CBS with feature extraction networks (FEN) and graph networks, utilizing temperature normalization (T-Norm) for model calibration.

Main Results:

• The incremental DG model successfully adapted to the target domain (transoral robotic surgery) while maintaining performance in the source domain (nephrectomy surgery).
• The E-CBS and T-Norm enhanced graph model outperformed state-of-the-art models in instrument-tissue interaction detection.
• Incremental DG demonstrated superior performance compared to naive domain adaptation and standard DG techniques.

Conclusions:

• The proposed incremental DG method effectively enhances robotic surgery scene understanding and instrument-tissue interaction detection.
• The approach allows robotic systems to generalize to new surgical domains while retaining knowledge from previous ones.
• The integration of E-CBS and T-Norm significantly improves the robustness and accuracy of surgical scene analysis models.