Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep...

Area of Science:

• Computer Vision
• Medical Robotics
• Surgical Technology

Background:

• Minimally invasive surgeries (MIS) face challenges due to indirect visualization and instrument manipulation.
• Accurate monitoring of surgical instruments is crucial but difficult in MIS.
• Vision-based tracking offers a flexible, software-based solution without altering surgical instruments or workflow.

Purpose of the Study:

• To develop and validate a robust 2D/3D tracking-by-detection framework for surgical instruments in MIS.
• To address the limitations of indirect observation and manipulation in MIS.

Main Methods:

• A novel 2D/3D tracking-by-detection framework is proposed for MIS instruments.
• Shaft tracking utilizes line features with the RANSAC algorithm.
• End-effector tracking employs deep learning with a convolutional neural network for specialized image features.

Main Results:

• Experimental verification in 2D and 3D using ex-vivo video sequences.
• Qualitative validation on in-vivo video sequences.
• The proposed method demonstrates robust and accurate tracking performance.

Conclusions:

• The 3D tracking performance in ex-vivo sequences surpasses current state-of-the-art methods.
• The method successfully tracks instruments even with unknown camera parameters in in-vivo sequences.
• Future work will focus on addressing occlusion and multi-instrument scenarios in MIS.