Related Experiment Video
Updated: May 20, 2026
Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality
Published on: August 9, 2024
A Granular Interaction Framework for Surgical Robotic Digital Twins: Enabling Realistic Segment-Level Manipulation in
An-Nhien Vo1,2, Dhanakumaresh Subramani3,4, Joachim Schmidt3,4,5
1Department of Thoracic Surgery, University Hospital Bonn, Bonn, Nordrhein-Westphalia, Germany. an.vo@ukbonn.de.
Introduction:
While Robot-Assisted Surgery (RAS) offers significant clinical advantages, adoption is often hindered by steep learning curves. Extended Reality (XR) and Digital Twins (DT) provide promising solutions for surgical training. However, existing integration pipelines require high-fidelity system-level simulations that address complex kinematic constraints. This paper proposes a modernized framework for integrating RAS into Extended Reality (XR) by transitioning from CAD models to scan-derived digital twins (DTs), as well as a force-based interaction paradigm for realistic simulation.
Methods:
Performance profiling was conducted on the Apple Vision Pro (AVP) to establish empirical hardware benchmarks for polygon counts. Utilizing 3D scanning and manual retopology, we developed a digital surrogate of the Dexter Surgical Robotic System, prioritizing model fidelity within stand-alone XR constraints. Within Unity, we evaluated Rigidbody versus ArticulationBody physics for kinematic stability. This necessitated the development of a force-based interaction system supporting both linear feedback (PD) and geometric constraint-based controllers.
Results:
The scan-derived model achieved a spatial accuracy of 1.6mm relative to the physical system. ArticulationBody physics demonstrated superior numerical stability compared to traditional Rigidbody components. While the linear feedback controller showed higher quantitative precision, it required extensive manual gain tuning; conversely, the geometric constraint-based controller provided high-speed stability without iterative calibration.
Conclusion:
This framework modernizes RAS integration by providing a non-CAD-dependent workflow, a prescriptive testing protocol for mesh decimation, and a force-based interaction system that enhances kinematic realism for clinical training.
