The surgeon in the loop: automation bias and human-AI interaction in robotic surgery
Related Experiment Video
Updated: Nov 8, 2026
A Teleoperated Robotic System-Assisted Percutaneous Transiliac-Transsacral Screw Fixation Technique
Published on: January 6, 2023
Maria Clara Morais1,2,3, Aditya Amit Godbole4,5,6, Emaad Iqbal4,5,6
1Intraoperative Performance Analytics Laboratory, Department of Surgery, Lenox Hill Hospital, Northwell Health, New York, NY, USA. drmariamorais@gmail.com.
Introduction:
Artificial intelligence systems for intraoperative surgical guidance are approaching clinical deployment, yet no study has quantified automation bias (defined as the tendency to over-rely on automated recommendations) in the context of surgical decision-making. As these systems enter operative workflows, the surgeon's ability to recognize and override incorrect AI guidance becomes a critical safety requirement.
Aim:
To measure the extent to which surgeons modify incision plans after exposure to AI-labeled visual guidance and to determine the rate at which incorrect AI suggestions are accepted.
Methods:
A two-phase controlled experiment was conducted among 19 surgeons (10 residents, 2 fellows, 7 attendings). Participants drew incision lines on 20 static frames from robotic cholecystectomy videos, first without guidance (Phase 01) and then with a mock AI overlay (Phase 02). Overlays were either correct (derived from expert consensus) or deliberately incorrect (shifted toward unsafe anatomy) at an approximately 70:30 ratio. No actual AI model was used. The primary outcome was mean perpendicular distance (MPD) between Phase 01 and Phase 02 lines. Secondary outcomes included AI-corridor adherence, over-reliance and under-reliance rates, behavioral accuracy, and confidence shift.
Results:
Across 231 paired observations, MPD did not differ significantly between correct (75.5 pixels; SD, 33.4) and incorrect (71.1 pixels; SD, 31.7) overlay conditions (P = .35). Directional analysis revealed that surgeons accepted incorrect AI suggestions in 60.6% of incorrect-overlay frames (43/71), with a specificity for rejecting incorrect guidance of only 39.4% (28/71). Correct suggestions were accepted 76.9% of the time (123/160). Overall behavioral accuracy was 65.4%. Self-reported confidence increased significantly after AI exposure (Δ = + 0.65; P < .001).
Conclusion:
Surgeons accepted approximately 3 of every 5 incorrect AI recommendations. As intraoperative AI systems approach deployment, surgeon oversight cannot be assumed to neutralize algorithmic error; it must be empirically measured, deliberately designed for, and prospectively validated.
