Automatic and Real-Time Surgeon's Gazing Point Detection From Surgical Videos Using Machine Learning and Mathematical...

Application of artificial intelligence (AI) in intraoperative imaging has been expanding rapidly. The surgeon's gazing point indicates the exact site of surgical procedures and concentrates critical information for AI applications. This study aimed to develop a machine learning-based system to automatically detect the surgeon's gazing point from surgical video data.

Surgical instruments were detected using a deep-learning model applied to images extracted from pancreaticoduodenectomy videos. Gazing points were estimated through a mathematical algorithm based on the axes and intersections of detected instruments, and time-averaging was applied to enhance stability in real-time analysis. After validation using pancreaticoduodenectomy cases, the system was subsequently applied to extended cholecystectomy and distal pancreatectomy cases to evaluate its applicability to other procedures.

Surgical instrument detection yielded AP50 of 60.5%. Gaze points detection achieved accuracies of 82.7% and 93.9% within 216- and 324-pixel radii (9.42% and 21.2% of a 1440 × 1080 screen) in pancreaticoduodenectomy. When applied to extended cholecystectomy and pancreaticoduodenectomy distal pancreatectomy, our system demonstrated comparable performance, with an accuracy of 85.5% within the 324-pixel radius. Time averaging improved accuracy, particularly with a 5-s average.

Our system successfully detected the surgeon's gaze point across procedures, suggesting potential utility in future AI-assisted surgery.