Visual marker-based augmented reality for intraoperative ultrasound in minimally invasive surgery

Intraoperative ultrasonography (IOUS) is essential in minimally invasive hepatobiliary-pancreatic surgery but requires surgeons to alternate attention between laparoscopic and ultrasound monitors, disrupting workflow. We developed Flag AR, an augmented reality system that superimposes real-time ultrasound images onto laparoscopic displays using visual marker tracking.

This prospective observational study included 14 patients undergoing minimally invasive hepatobiliary-pancreatic procedures between March and April 2025. Three experienced surgeons performed target identification tasks using both Flag AR and conventional IOUS. Eye-tracking analysis measured gaze shifts and visual focus time using wearable eye-tracking glasses.

Twenty-six anatomical targets were evaluated, including liver tumors (n = 13), gallbladder tumors (n = 4), and vascular structures (n = 9). Although task completion time showed no significant difference between modalities (23.8 vs. 22.1 s, P = 0.351), Flag AR significantly reduced gaze shifts compared to conventional IOUS (1.8 vs. 8.4 shifts, P < 0.001) and increased time focused on a single monitor (97.9% vs. 73.8%, P < 0.001).

Flag AR represents the first clinical application of purely visual marker-based ultrasound overlay technology in minimally invasive surgery. By enabling sustained visual focus on a single display, this system improves surgeon workflow efficiency during IOUS procedures without compromising task performance, potentially enhancing spatial visualization and reducing cognitive workload in complex surgical procedures.