Evaluating the Accuracy of ChatGPT-4o in Addressing Complex Clinical Questions Based on NCCN Guidelines for Rectal
Ryan Meyer1, Tamir E Bresler1, Kevin M Palmer1
1Department of Surgery, Los Robles Regional Medical Center, Thousand Oaks, California, USA.
Introduction:
The management of rectal adenocarcinoma requires navigation of complex, branching guideline pathways encompassing neoadjuvant sequencing, surgical approach, organ preservation, and surveillance, yet real-world guideline adherence remains as low as 60-70%. The ability of current-generation large language models (LLMs) to accurately navigate these decision points has not been fully characterized.
Methods:
In this cross-sectional, vignette-based study, 135 clinical questions were constructed from 45 pages of NCCN Rectal Cancer Guidelines (Version 4.2024). ChatGPT-4o was queried using standardized prompts with up to 3 clarifying questions permitted per query. Responses were independently evaluated by two physician raters on a 5-point Likert scale, with potential discrepancies adjudicated by a board-certified surgical oncologist. Primary outcomes were the proportion of responses rated Correct (score ≥ 3) and Accurate (score ≥ 4). Inter-rater reliability was assessed using Cohen's kappa, and subgroup analysis was performed across clinical domains using the Kruskal-Wallis test.
Results:
Of 135 questions, 127 (94.1%; 95% CI, 88.7-97.0%) were Correct and 121 (89.6%; 95% CI, 83.3-93.7%) were Accurate. One hundred two responses (75.6%) were completely correct without additional prompting. Performance was consistent across clinical domains (Kruskal-Wallis H = 0.530, p = 0.767). Inter-rater agreement was perfect (κ = 1.0). Eight responses (5.9%) contained partially or wholly incorrect information, with errors concentrated in multi-step conditional treatment decision points.
Conclusion:
ChatGPT-4o demonstrates high concordance with NCCN rectal cancer guidelines across all evaluated clinical domains with notable improvement over prior ChatGPT iterations evaluated by our group. The concentration of errors in complex conditional treatment algorithms suggests that LLMs excel at discrete factual recall but may struggle with multi-step reasoning under clinical uncertainty. Prospective validation using real-world clinical data and comparison with multidisciplinary tumor board recommendations remain necessary prior to clinical integration.
Insights
Large language models like ChatGPT-4o show high accuracy in following rectal cancer guidelines, demonstrating improved performance over previous versions. However, complex treatment decisions remain a challenge for these AI tools.
Area of Science:
- Oncology
- Artificial Intelligence
- Medical Guidelines
Background:
- Rectal adenocarcinoma management involves complex guidelines with variable adherence.
- Current large language models' (LLMs) ability to navigate these guidelines is not well-understood.
Purpose of the Study:
- To evaluate the accuracy of ChatGPT-4o in adhering to NCCN Rectal Cancer Guidelines.
- To assess LLM performance across different clinical domains within rectal cancer management.
Main Methods:
- A cross-sectional study using 135 clinical questions derived from NCCN guidelines.
- ChatGPT-4o was queried, and responses were rated by physician reviewers.
- Inter-rater reliability and performance across domains were analyzed.
Main Results:
- ChatGPT-4o achieved 94.1% Correct and 89.6% Accurate responses.
- Performance was consistent across clinical domains, with perfect inter-rater agreement.
- Errors were concentrated in complex, multi-step treatment decision points.
Conclusions:
- ChatGPT-4o demonstrates high concordance with NCCN rectal cancer guidelines.
- LLMs excel at factual recall but may struggle with complex clinical reasoning.
- Further validation is needed before clinical integration.