Perception-first digital twin for augmented reality microsurgery

Augmented reality guidance in microsurgery is challenged by depth ambiguity, occlusion, and limited situational awareness under the operating microscope. Digital twins, dynamic virtual models of physical systems, can provide the contextual geometric data needed to mitigate these limitations. We introduce a perception-first digital twin framework that uses real-time surgical state to drive depth-, occlusion-, and proximity-aware augmented reality visualization for guidance in otologic surgery.

We formulate a hand-eye calibration refinement as a joint $\text{SE}\left(3\right)$ optimization with a bias term and robust weighting to ensure accurate alignment between physical and virtual entities. Our framework integrates this optimized calibration with high-precision optical tracking of key surgical elements and real-time simulation, enabling dynamic updates of the digital twin from the evolving surgical state. In turn, the updated virtual replica is used to compute perceptual fields that drive the augmented reality overlays on the live stereoscopic microscope feed of a 3D-printed anatomical model.

We evaluated the accuracy of the optimized calibration on both calibration and evaluation datasets. The optimized workflow achieved a median translational error of $1.06\text{mm}$ and $1.19\text{mm}$ and a median rotational error of $0.{46}^{\circ }$ and $0.{28}^{\circ }$ , respectively. In addition, the digital twin was used to derive and overlay perceptual cues onto the microscope video of a 3D-printed temporal bone. An AR-guided microsurgical use case demonstrated how the dynamically updated cues convey spatial relationships between critical structures and the evolving surgical cavity.

We present a perception-first digital twin framework for augmented reality guidance in microsurgery. As surgical alteration of anatomy progresses, the digital twin is continuously updated and used to drive perception-aware augmented reality cues. Our framework enables enhanced spatial understanding while preserving visibility of the operative field.