Augmenting Large Language Models With National Comprehensive Cancer Network Guidelines for Improved and Standardized...

Area of Science:

• Oncology
• Artificial Intelligence
• Medical Informatics

Background:

• Multidisciplinary breast tumor boards (MTBs) are crucial for breast cancer treatment optimization.
• Challenges in MTBs include logistics, variable expertise, and lack of standardization.
• Artificial intelligence (AI) and large language models (LLMs) offer potential solutions for clinical decision support.

Purpose of the Study:

• To evaluate the accuracy of adjuvant therapy recommendations from an AI tool, TheSerenityBot (TSB).
• To compare TSB's recommendations against Claude-2 and GPT-4.
• To use expert MTB consensus as the reference standard for accuracy.

Main Methods:

• Retrospective analysis of 50 postoperative breast cancer cases from National University Hospital, Singapore (June-November 2023).
• TSB, an AI model based on Claude-2 and updated NCCN guidelines, generated recommendations.
• Model performance was assessed by comparing AI outputs with MTB recommendations using generalized estimating equations.

Main Results:

• TSB achieved the highest overall accuracy (0.89), followed by Claude-2 (0.86) and GPT-4 (0.78).
• GPT-4 showed significantly lower accuracy for genetic testing recommendations (OR, 0.05; P < .001).
• Claude-2 demonstrated lower accuracy in radiotherapy recommendations (OR, 0.41; P = .040).

Conclusions:

• Guideline-augmented AI tools like TSB show promise in supporting breast cancer adjuvant therapy decisions.
• Future AI iterations aim to enhance clinical relevance by incorporating patient factors and EHR integration.
• Ongoing prospective trials will assess the real-world impact of AI in breast cancer treatment planning.