Real-time marker-less needle tracking for CT-guided interventions using multiple RGB cameras

In CT-guided interventions, instrument tracking reduces patient risk while improving the overall outcome. Though useful, marker-based tracking systems suffer from occlusion, are complex to set up, and fail when needles deflect. A marker-less tracking system could overcome these challenges, enabling streamlined integration.

We developed and evaluated a proof-of-concept system combining standard RGB cameras with a U-Net-ConvNeXt architecture for needle detection and tracking. We compiled a fully annotated dataset of 35,000 frames, spanning multiple needle types and scenarios. We evaluated tracking accuracy against state-of-the-art marker-based systems, utilizing an IR-based and an image-based system on static positioning, dynamic movement, and needle-deflection scenarios. Moreover, our marker-less system was evaluated using both known and unknown needle types.

Our proof-of-concept system demonstrated a 3D tracking error of $2.70\pm 1.07$  mm at 25 FPS with 3 cameras, outperforming ArUco marker-based tracking ( $5.29\pm 1.85$  mm). Tracking accuracy remained consistent for unknown needle types not included in the training data, suggesting strong generalization. The system maintained a 99.4 % needle-tip detection rate across all frames with robust performance during needle deflection, while marker-based systems failed under identical conditions. With only 2 cameras, the tracking error of our system increased modestly ( $4.81\pm 2.38$  mm).

Our proof-of-concept demonstrates the feasibility of marker-less needle tracking, suggesting potential for clinically relevant accuracy, reliability, and reduced complexity while exhibiting robustness to needle deflection and varying needle appearances. These preliminary results indicate promising directions for future clinical development.