In Vivo Demonstration of Deep Learning-Based Photoacoustic Visual Servoing System

To develop the first known deep learning-based photoacoustic visual servoing system utilizing point source localization and hybrid position-force control to track catheter tips in three dimensions in real-time.

We integrated either object detection or instance segmentation-based localization with hybrid position-force control to create our novel system. Cardiac catheter tips were then tracked across distances of 40 mm in a plastisol phantom and 25-64 mm in an in vivo swine in real-time in nine visual servoing trials total.

Object detection-based localization identified the cardiac catheter tip in 88.0-91.7% and 66.7-70.4% of phantom and in vivo channel data frames, respectively. Instance segmentation detection rates ranged 86.4-100.0% in vivo. These catheter tips were tracked with errors as low as 0.5 mm in phantom trials and 0.8 mm in the in vivo trials. The mean inference times were $\geq$ 145.3 ms and $\geq$ 516.3 ms with object detection-based and instance segmentation-based point source localization, respectively. The hybrid position-force control system enabled contact with the imaging surface during $\geq$99.43% of each visual servoing trial.

Our novel deep learning-based photoacoustic visual servoing system was successfully demonstrated. Object detection-based localization operated with inference times that are more suitable for real-time implementations while instance segmentation had lower tracking errors.

After implementing suggested optimization modifications, our novel system has the potential to track catheter tips, needle tips, and other surgical tool tips in real-time during surgical and interventional procedures.