STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

Area of Science:

• Medical technology
• Computer-assisted surgery
• Augmented Reality (AR)

Background:

• Augmented Reality (AR) aids surgeons by providing visual navigation during procedures.
• Current AR systems often rely on external infrared cameras and markers for tracking surgical tools and patient pose.
• Existing medical-grade tracking systems require specialized hardware and integration into the operating room environment.

Purpose of the Study:

• To develop and evaluate a framework enabling accurate tracking of surgical tools using only the built-in cameras of AR Head-Mounted Displays (HMDs).
• To eliminate the need for additional electronics or external tracking systems within the HMD for marker-based pose estimation.
• To demonstrate the framework's capability for simultaneous multi-tool tracking without prior geometric knowledge.

Main Methods:

• A novel framework utilizing the integrated cameras of AR HMDs for retro-reflective marker tracking was developed.
• The system establishes a local network connection between the AR headset and a workstation for data processing.
• Performance was evaluated through marker tracking accuracy measurements and simulated orthopedic k-wire insertion procedures with surgeons.

Main Results:

• The framework achieved high accuracy in marker tracking: 0.09±0.06 mm (lateral), 0.42±0.32 mm (longitudinal), and 0.80±0.39° (rotational).
• Surgical simulations involving k-wire insertions demonstrated the system's practical applicability and effectiveness in a clinical context.
• Accuracy in surgical use cases was found to be comparable to existing AR-based navigation systems reported in the literature.

Conclusions:

• The proposed framework successfully enables accurate and reliable tracking of surgical tools using only AR HMD built-in cameras.
• This approach offers a cost-effective and integrated solution for AR-assisted surgical navigation, reducing hardware complexity.
• The system shows significant potential for enhancing precision and safety in various image-guided surgical interventions.