SurgiTrack: Fine-grained multi-class multi-tool tracking in surgical videos

Area of Science:

• Medical Imaging
• Computer Vision
• Surgical Robotics

Background:

• Accurate surgical tool tracking is crucial for computer-assisted interventions.
• Existing methods struggle with dynamic scenarios like out-of-body views and tool re-identification due to visual similarity and lack of operator information.

Purpose of the Study:

• To develop a novel deep learning method for robust surgical tool tracking that addresses limitations of previous approaches.
• To improve tool re-identification in challenging scenarios, including occlusion and re-insertion, by incorporating operator-related information.

Main Methods:

• Proposed SurgiTrack, a deep learning method utilizing YOLOv7 for tool detection and an attention mechanism to model tool originating direction as a proxy for operators.
• Introduced a harmonizing bipartite matching graph to handle diverse tool trajectory perspectives and ensure accurate identity association.
• Utilized the CholecTrack20 dataset with fine-grained labels for intraoperative, intracorporeal, and visibility perspectives.

Main Results:

• SurgiTrack demonstrated superior performance compared to baseline and state-of-the-art methods on the CholecTrack20 dataset.
• The method achieved real-time inference capability, crucial for practical surgical applications.
• Successfully addressed the challenge of re-identifying tools with high visual similarity, especially within the same category.

Conclusions:

• SurgiTrack offers a significant advancement in surgical tool tracking, providing dynamic trajectories for enhanced computer-assisted interventions.
• The novel approach of using operator direction as a proxy improves tool re-identification accuracy in complex surgical video scenarios.
• This work establishes a new benchmark for adaptable and precise tool tracking in minimally invasive surgery.