Automatic and Real-Time Surgeon's Gazing Point Detection From Surgical Videos Using Machine Learning and Mathematical Algorithm

Shu Sasaki ¹, Kenji Karako ¹, Kyoji Ito ¹

⁰Hepato-Biliary-Pancreatic Surgery Division, Department of Surgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.

Journal of Hepato-Biliary-Pancreatic Sciences +

|

December 19, 2025

View abstract on PubMed

Summary

This study developed an AI system to automatically detect a surgeon's gaze point from surgical videos. The system shows promise for enhancing AI-assisted surgical procedures.

Area Of Science

Medical Artificial Intelligence
Surgical Technology
Computer Vision in Surgery

Background

Artificial intelligence (AI) is increasingly used in intraoperative imaging.
The surgeon's gaze point is crucial for identifying surgical sites and providing data for AI.
Current AI applications require precise identification of the surgeon's focus.

Purpose Of The Study

To develop a machine learning system for automatic detection of the surgeon's gaze point using surgical video data.
To validate the system's performance across different surgical procedures.
To assess the potential utility of gaze point detection in AI-assisted surgery.

Main Methods

Surgical instruments were detected using a deep-learning model on video frames.
Gaze points were calculated via a mathematical algorithm based on instrument axes and intersections.
Time-averaging was employed for real-time analysis stability and validated on pancreaticoduodenectomy, extended cholecystectomy, and distal pancreatectomy cases.

Main Results

Surgical instrument detection achieved an AP50 of 60.5%.
Gaze point detection accuracy reached 82.7% and 93.9% within specified pixel radii for pancreaticoduodenectomy.
The system demonstrated comparable performance (85.5% accuracy within a 324-pixel radius) in other procedures, with time-averaging improving results.

Conclusions

The developed system successfully detected surgeon gaze points across multiple surgical procedures.
This technology holds potential for integration into future AI-assisted surgical systems.
Accurate gaze point detection is a key step towards more sophisticated AI surgical tools.

Keywords:

artificial intelligence deep learning pancreaticoduodenectomy video recording