Neural digital twins: reconstructing complex medical environments for spatial planning in virtual reality

Area of Science:

• Medical technology
• Virtual reality
• Computer vision

Background:

• Operating room (OR) complexity is increasing due to specialized robotic and surgical tools.
• Optimizing OR space utilization for new techniques is challenging due to limited physical availability.
• Digital twins are needed for immersive spatial planning in virtual reality (VR).

Purpose of the Study:

• To present a neural rendering-based method for creating immersive digital twins of medical environments and devices.
• To enable spatial planning of surgical scenarios using these digital twins.
• To evaluate the feasibility and benefits of this approach for complex OR planning.

Main Methods:

• Developed a neural rendering technique to generate digital twins from casual video capture.
• Reconstructed two operating rooms and ten objects using neural reconstruction.
• Conducted a user study with 21 participants performing planning tasks in VR environments.

Main Results:

• Neural reconstruction significantly increased perceived utility and presence compared to low-complexity versions.
• Users exhibited higher perceived workload and exploratory behavior in the neural reconstruction environments.
• No significant difference in interactivity was observed between the environments.

Conclusions:

• Modern reconstruction techniques can create digital twins of complex medical environments and objects.
• Users can create, explore, and interact with virtual objects without expert knowledge or specialized hardware.
• The approach shows promise for spatial planning due to high perceived utility and technical accessibility.