ChatGPT vs. surgeons on pancreatic cancer queries: accuracy & empathy evaluated by patients and experts

Artificial intelligence (AI) offers potential support in patient-clinician interactions, but its impact on such communication remains unexplored.

In this study, ChatGPT was compared with two pancreatic surgeons in responding to ten pancreatic cancer surgery-related questions, co-designed with the Patient Advisory Board of the Surgical Society's Study Center. A blind evaluation of these responses, considering content congruency and clarity for non-specialists, was conducted by patients and surgeons.

From June 23 to July 21, 2023, 24 patients and 25 surgeons participated, of which eleven patients and ten surgeons completed the survey in full. Utilizing a quantitative scale from 1 (strong-disagreement) to 5 (full-agreement), consensus was observed among patients and specialists concerning the content delivered by ChatGPT. The metrics for comprehensibility to a non-specialist audience consistently showed positive reception. In the evaluation of empathetic resonance, ChatGPT's responses mirrored those of the surgeons in the patient's view. A significant proportion ranked Surgeon 1's contributions foremost, followed closely by ChatGPT.

This study demonstrates that surgeons and ChatGPT answer common queries from patients regarding pancreatic cancer surgery comparable regarding reliability, lay comprehension and empathy as evaluated by patients and surgical experts. These findings highlight the potential of AI in enhancing patient-provider interactions.