Deep learning-driven multi-hierarchical granularity integration for surgical scene understanding: experimental study.

Area of Science:

• Computer vision in surgery
• Artificial intelligence in medicine
• Surgical data science

Background:

• Computer comprehension of surgical scenes is vital for intelligent surgical assistance.
• Surgical scene information includes both overall process (coarse-grained) and specific operations (fine-grained) data.
• Standardized datasets and advanced AI are needed to support clinical applications.

Purpose of the Study:

• To construct a standardized, full-grained annotated dataset for laparoscopic radical nephrectomy.
• To develop a deep learning framework for integrating multi-hierarchical granularity information.
• To support clinical intelligent applications through enhanced surgical scene understanding.

Main Methods:

• Collected 41 multi-center videos and performed comprehensive annotations (phases, steps, instrument segmentation, surgical action triplets).
• Designed a lightweight deep learning framework for simultaneous multi-granularity information perception.
• Developed an intraoperative bleeding prediction model and an automated postoperative surgical summary report.

Main Results:

• The dataset contains 141,443 frames with phase/step annotations, 8,435 with instrument segmentation, and 25,305 with surgical action triplets.
• The deep learning framework outperformed state-of-the-art algorithms on various granularity tasks.
• Key surgical steps involving renal artery, vein, and ureter were identified as critical predictors of bleeding risk.

Conclusions:

• Developed the first fully annotated dataset for laparoscopic radical nephrectomy.
• Created a novel framework for bidirectional enhancement of coarse and fine-grained surgical information.
• Laid a foundation for advancing intelligent surgery via a data-algorithm-application cycle.