Real-time tracking of surgical instruments based on spatio-temporal context and deep learning

Area of Science:

• Medical Imaging
• Computer-Assisted Interventions
• Surgical Robotics

Background:

• Real-time tool tracking is crucial for computer-assisted interventions (CAIs) in minimally invasive surgery (MIS).
• Existing visual tracking methods often fail due to motion blur, poor lighting, reflections, shadows, and occlusions.

Purpose of the Study:

• To develop an automatic, real-time 2D surgical tool detection and tracking method.
• To improve the accuracy and speed of visual tracking in challenging surgical scenarios.

Main Methods:

• Utilized a convolutional neural network (CNN) integrated with a spatial transformer network (STN) and spatio-temporal context (STC).
• Trained the STN in-house to enhance tool localization accuracy and speed.
• Compared the proposed method against four other CAI visual tracking methods.

Main Results:

• The proposed method demonstrated superior performance in both accuracy and speed compared to existing algorithms.
• Achieved real-time tracking of surgical tools without requiring labels, even in difficult conditions.
• Performance was validated across eight diverse datasets, including in vivo abdominal, cardiac, and retinal cases.

Conclusions:

• The developed method offers a robust solution for real-time surgical tool tracking in MIS.
• It provides accuracy comparable to or exceeding state-of-the-art methods.
• Future work will address challenges related to occlusion and multi-instrument tracking.