How soon will surgeons become mere technicians? Chatbot performance in managing clinical scenarios

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Surgical Education

Background:

• Chatbots are increasingly used in medicine for clinical decision support.
• Concerns exist regarding the accuracy of information provided by artificial intelligence (AI) platforms.
• This study evaluates the performance of popular chatbots against expert surgeons.

Purpose of the Study:

• To assess the accuracy and performance of leading AI chatbots in a medical context.
• To compare the diagnostic, evaluative, and treatment-related responses of chatbots with those of board-certified thoracic surgeons.
• To identify potential risks associated with AI implementation in clinical decision-making.

Main Methods:

• Developed clinical scenarios based on the American Board of Thoracic Surgery (ABTS) Qualifying Exam.
• Utilized the Key Feature methodology for scenario construction, focusing on diagnosis, evaluation, and treatment.
• Administered 10 scenarios to four chatbots (ChatGPT-4, Gemini, Perplexity, Claude 2) and 21 board-certified surgeons, comparing scores using the Mann-Whitney U test.

Main Results:

• Chatbots achieved a median score of 1.06 per scenario, significantly lower than surgeons' median score of 1.88 (P=.019).
• Surgeons outperformed chatbots in all but two scenarios.
• Chatbot responses were more likely to result in critical failures (median 0.50) compared to surgeon responses (median 0.19; P=.016).

Conclusions:

• Popular AI chatbots demonstrate significantly lower performance levels compared to board-certified thoracic surgeons.
• The findings underscore the need for caution when implementing AI in clinical decision-making processes.
• Further research is warranted to improve AI accuracy and safety in medical applications.