Extended Reality for Enhancing Surgery: Research Directions to Transform the Operating Environment

In the past decade, virtual reality (VR) and augmented reality (AR) have seen a second wave of activity due to consumer-level availability of devices. In the medical realm, use of VR and AR has been investigated extensively and successfully for training, teaching, rehabilitation, and therapy. However, extended reality (XR) and its intraoperative use in the operating room has yet been underexplored and underdeveloped. Intelligent XR environments that integrate advanced visualization, intuitive interaction, and AI-assisted decision support have the potential to transform clinical care. Responsive, context-aware spaces could enhance medical teams' capabilities by enabling real-time access to critical information, enhancing situation awareness, and supporting remote collaboration and access to clinical expertise across geographical boundaries-all while preserving the natural flow of procedures. But to realize the potential benefits of XR in the operating room requires a significant and focused effort by an interdisciplinary community. It is the intention of this paper to help to catalyze such an effort. In this paper, we identify pressing issues in clinical care, the potential of XR to address them, and the further research that is needed to realize that potential. We look at a broad range of areas where XR can play an important role, including direct support for the surgeon, support for the surgical team, support for the patient, and remote collaboration. We also consider ways in which XR can be integrated with AI to provide enhanced information and interaction. Discussions are underpinned by considerations of methods to evaluate contributions and progress. This paper is the result of a Dagstuhl Seminar on Extended Reality for the Operating Room (XR4OR), held at the Leibniz Center for Informatics, Schloss Dagstuhl, Germany, in February 2025. It gathered twenty-four researchers working in the areas of virtual reality, augmented reality, medical informatics, human-computer interaction, and surgery.