Evaluating the Accuracy of ChatGPT-4o in Addressing Complex Clinical Questions Based on NCCN Guidelines for Rectal
Ryan Meyer1, Tamir E Bresler1, Kevin M Palmer1
1Department of Surgery, Los Robles Regional Medical Center, Thousand Oaks, California, USA.
Introduction:
The management of rectal adenocarcinoma requires navigation of complex, branching guideline pathways encompassing neoadjuvant sequencing, surgical approach, organ preservation, and surveillance, yet real-world guideline adherence remains as low as 60-70%. The ability of current-generation large language models (LLMs) to accurately navigate these decision points has not been fully characterized.
Methods:
In this cross-sectional, vignette-based study, 135 clinical questions were constructed from 45 pages of NCCN Rectal Cancer Guidelines (Version 4.2024). ChatGPT-4o was queried using standardized prompts with up to 3 clarifying questions permitted per query. Responses were independently evaluated by two physician raters on a 5-point Likert scale, with potential discrepancies adjudicated by a board-certified surgical oncologist. Primary outcomes were the proportion of responses rated Correct (score ≥ 3) and Accurate (score ≥ 4). Inter-rater reliability was assessed using Cohen's kappa, and subgroup analysis was performed across clinical domains using the Kruskal-Wallis test.
Results:
Of 135 questions, 127 (94.1%; 95% CI, 88.7-97.0%) were Correct and 121 (89.6%; 95% CI, 83.3-93.7%) were Accurate. One hundred two responses (75.6%) were completely correct without additional prompting. Performance was consistent across clinical domains (Kruskal-Wallis H = 0.530, p = 0.767). Inter-rater agreement was perfect (κ = 1.0). Eight responses (5.9%) contained partially or wholly incorrect information, with errors concentrated in multi-step conditional treatment decision points.
Conclusion:
ChatGPT-4o demonstrates high concordance with NCCN rectal cancer guidelines across all evaluated clinical domains with notable improvement over prior ChatGPT iterations evaluated by our group. The concentration of errors in complex conditional treatment algorithms suggests that LLMs excel at discrete factual recall but may struggle with multi-step reasoning under clinical uncertainty. Prospective validation using real-world clinical data and comparison with multidisciplinary tumor board recommendations remain necessary prior to clinical integration.