Perception-first digital twin for augmented reality microsurgery

Area of Science:

• Medical Engineering
• Computer Science
• Surgical Technology

Background:

• Microsurgery augmented reality (AR) guidance faces challenges like depth ambiguity and limited situational awareness.
• Digital twins offer dynamic virtual models to provide crucial contextual geometric data.
• A perception-first digital twin framework is proposed to address these limitations in otologic surgery.

Purpose of the Study:

• To develop and introduce a perception-first digital twin framework for AR guidance in microsurgery.
• To leverage real-time surgical state for depth-, occlusion-, and proximity-aware AR visualization.
• To mitigate limitations of current AR systems under the operating microscope.

Main Methods:

• Formulated hand-eye calibration refinement using joint SE(3) optimization with robust weighting.
• Integrated optimized calibration with high-precision optical tracking and real-time simulation for dynamic digital twin updates.
• Utilized the updated digital twin to compute perceptual fields driving AR overlays on the microscope feed.

Main Results:

• Optimized calibration achieved median translational errors of 1.06 mm and 1.19 mm, and rotational errors of 0.46° and 0.28°.
• The digital twin successfully derived and overlaid perceptual cues onto a 3D-printed temporal bone model.
• An AR-guided microsurgical use case demonstrated enhanced spatial understanding of critical structures and the surgical cavity.

Conclusions:

• A perception-first digital twin framework for AR-guided microsurgery was successfully presented.
• The framework continuously updates the digital twin based on surgical progression to drive perception-aware AR cues.
• The system enhances spatial understanding while maintaining operative field visibility.