Real-time geometry-aware augmented reality in minimally invasive surgery

Area of Science:

• Medical Technology
• Computer Vision
• Surgical Navigation

Background:

• Minimally invasive surgery (MIS) presents challenges for real-time, accurate augmented reality (AR) information overlay.
• Existing AR systems struggle to balance computational performance and accuracy in complex surgical environments.

Purpose of the Study:

• To present a novel real-time AR framework for MIS that enables interactive, geometrically aware AR in endoscopic surgery.
• To improve the precision and accuracy of surgical guidance through enhanced visualization and data overlay.

Main Methods:

• Developed a framework for real-time tracking of endoscopic camera movement by minimizing re-projection error.
• Implemented a dense zero-mean normalized cross-correlation stereo-matching method for incremental 3D mesh reconstruction.
• Achieved intra-operative geometric information inference without pre-operative scans or templates.

Main Results:

• The framework successfully tracks camera movement with fast performance.
• Accurate 3D surface reconstruction of the MIS scene was achieved.
• Demonstrated improved precision in adding annotations, localizing tumors/vessels, and measurement labeling compared to state-of-the-art methods.

Conclusions:

• The proposed AR framework offers a significant advancement for real-time surgical guidance in MIS.
• The system's ability to provide accurate geometric information intra-operatively enhances surgical precision and safety.
• This technology has the potential to overcome current limitations in AR-assisted minimally invasive procedures.