AR in VR: assessing surgical augmented reality visualizations in a steerable virtual reality environment

Area of Science:

• Medical technology
• Surgical innovation
• Human-computer interaction

Background:

• Augmented reality (AR) shows promise for surgical support but faces limited real-world application.
• Standardized evaluation of AR surgical visualizations is lacking, hindering development.
• A flexible framework is needed to assess AR visualization techniques and scenarios.

Purpose of the Study:

• To introduce an interactive virtual reality (VR) framework, the Augmented Visualization Box (AVB), for systematically evaluating AR surgical visualizations.
• To enable rapid assessment of AR visualization techniques and parameter spaces without complex registration.
• To quantitatively benchmark AR overlay methods in simulated intraoperative scenarios.

Main Methods:

• Developed the Augmented Visualization Box (AVB) using a VR approach to simulate complex environments.
• Created a virtual laparoscopic scenario with anatomical models in a game engine.
• Conducted a study with eleven surgeons to assess VR environment factors and AR overlay concepts.

Main Results:

• Surgeons found 2D surface details and lighting conditions most impactful on VR scene perception.
• Depending on the AR visualization's depth encoding, 37% to 91% of surgeons made incorrect decisions but were confident in their accuracy.
• Surgeons exhibited increased confidence in their decisions when using AR visualizations, even when those decisions were wrong.

Conclusions:

• The AVB framework realistically simulates intraoperative situations for quantitative benchmarking of AR overlay methods.
• Effective surgical task execution with AR systems necessitates visualizations tailored to specific surgical tasks.
• Customizing AR visualizations is crucial for enhancing surgical performance and safety.