Vision-based object detection and tracking from surgical microscope video for augmented cochlear implant surgery

Cochlear implants (CIs) have revolutionized the treatment of severe to profound sensorineural hearing loss, with life-altering outcomes for patients. CI surgery requires the surgeon to delicately place an electrode array in the scala tympani of the cochlea, a procedure that often challenges surgeons due to the absence of visual guidance. In this work, we aim to develop a vision-based tracking strategy to track the movements of the patient and the surgical tools.

A comprehensive object detection and tracking workflow tailored for temporal bone environments was developed by evaluating the performance of multiple object detection models, which were integrated with a modified BoT-SORT algorithm to enable accurate identification, segmentation, and tracking of surgical tools, electrodes, the surgeon's hands, and the patient's head across frames. The tracking strategy was trained and tested on both in-vivo and ex-vivo datasets.

YOLOv9-based framework achieved the best performance with a mAP50 of 0.788 for detection and 0.762 for segmentation.

Our findings highlight the promise of vision-based detection and tracking models in surgical settings, while also underscoring the challenges of translating performance from ex vivo to in vivo environments.

The integrated YOLOv9 and BoT-SORT tracking system maintained stable identity preservation and trajectory consistency, demonstrating strong potential for real-world surgical applications.