From microscope to head-mounted display: integrating hand tracking into microsurgical augmented reality.

Area of Science:

• Medical Technology
• Surgical Navigation
• Human-Computer Interaction

Background:

• Operating microscopes are crucial for otologic microsurgery, requiring precise navigation in confined spaces.
• Augmented reality (AR) integrates preoperative imaging (CT) with live microscope feeds for enhanced surgical guidance.
• Current AR systems necessitate manual computer interaction, disrupting surgical flow and sterility.

Purpose of the Study:

• To implement and evaluate free-hand interaction using hand tracking and gesture recognition for AR surgical guidance.
• To reduce disruption during surgery and improve human-computer interaction in otologic microsurgery.

Main Methods:

• Calibrated an electromagnetically tracked surgical microscope with a custom 3D printed board.
• Integrated Ultraleap's Leap Motion Controller 2 for hand tracking and gesture recognition.
• Collected surgeon feedback and conducted user tasks involving virtual model interaction and AR visualization adjustment.

Main Results:

• Enhanced hand interaction functionalities based on initial observations and user feedback.
• Users preferred the new interaction concepts for their minimal workflow disruption and intuitive control.
• The system demonstrated improved interaction with virtual content during simulated surgical tasks.

Conclusions:

• Successfully integrated head-mounted display (HMD)-style hand interaction concepts into a surgical microscope for AR otologic microsurgery.
• The developed system offers a more favorable human-computer interaction approach in surgical settings.
• This technology shows potential for more efficient surgical task execution under microscopic AR guidance.