Automatic Annotations by Augmented Reality-Enabled Laparoscopic Surgery.

Area of Science:

• Medical Image Computing
• Surgical Navigation
• Augmented Reality (AR)

Background:

• Accurate surgical procedure labels (e.g., image segmentations) are crucial for medical image computing tasks.
• Manual creation of these labels is time-consuming, costly, and requires expert involvement.
• Existing methods often struggle with occluded views or tools interacting outside the camera's field of view.

Purpose of the Study:

• To propose and evaluate an automated pathway for generating surgical procedure labels.
• To leverage spatial and temporal registration within augmented reality (AR) systems for label generation.
• To reduce the manual effort and cost associated with creating medical imaging datasets.

Main Methods:

• Utilizing spatial and temporal registration between patient data, anatomical models, tracked instruments, and the surgeon's view.
• Collecting comprehensive raw data (video, tracking, calibrations) from a live laparoscopic AR system.
• Converting surgical recordings into various representations to generate datasets as by-products.
• Employing 3D registration to overlay real objects with virtual counterparts for accurate labeling.

Main Results:

• The proposed system can generate valuable datasets as by-products of AR-enabled surgeries.
• The approach successfully creates labels even when visual input is occluded or tools are out of view.
• An interactive system was developed to assist users in annotating surgical data.
• Participant feedback indicated the system's efficacy and user-friendliness.

Conclusions:

• Automated generation of surgical labels is feasible using AR systems and 3D registration.
• This method significantly reduces the reliance on manual annotation by medical experts.
• The system offers a cost-effective and efficient solution for creating large-scale medical imaging datasets.
• Further development can enhance AR systems' utility in surgical data annotation and analysis.