Deep learning-driven multi-hierarchical granularity integration for surgical scene understanding: experimental study

A comprehensive understanding of surgical scenes by computers is a crucial foundation for achieving intelligent surgical assistance and autonomous decision-making. Surgical scene information encompasses coarse-grained data reflecting the overall process and fine-grained details showcasing specific operations. This study aims to construct a standardized, full-grained annotation dataset for laparoscopic radical nephrectomy and develop a deep learning framework for multi-hierarchical granularity integration, providing support for clinical intelligent applications.

We collected 41 multi-center videos and performed comprehensive annotations, including 8 surgical phases, 17 surgical steps, 6 types of instrument segmentation, and 35 surgical action triplets (SATs). Using this dataset, we designed a lightweight framework that enables simultaneous perception of multi-granularity information through shared feature extraction and task-specific decoders. Based on the comprehensive understanding information, we developed a robust intraoperative bleeding prediction model using 123 combinations of machine learning algorithms and generated an automated postoperative surgical summary report.

The dataset of this study comprises 141 443 frames with annotations for surgical phases and steps, 8435 frames with instrument segmentation annotations, and 25 305 frames with SATs. Our deep learning framework achieves outstanding performance across tasks of different granularities, with recognition accuracy even surpassing that of six state-of-the-art algorithms developed separately for individual subtasks. Based on full-granularity features, we found that surgical steps involving key anatomical structures such as the renal artery, renal vein, and ureter are critical predictors of bleeding risk. And the post-operative automated reporting system accurately extracts key surgical time points and specific procedural durations.

We constructed the first fully annotated dataset for laparoscopic radical nephrectomy, and developed a novel framework that achieves bidirectional compensation and enhancement of both coarse and fine-granularity information. This study not only expands the theoretical boundaries of artificial intelligence in surgical procedures, but also lays a solid foundation for advancing intelligent surgery through the "data-algorithm-application" cycle.