Augmenting Large Language Models With National Comprehensive Cancer Network Guidelines for Improved and Standardized...

Multidisciplinary breast tumor boards (MTBs) are essential for optimizing breast cancer treatment but face challenges related to logistics, variability in expertise, and lack of standardization. Large language models may support clinical decision making. This study evaluates the accuracy of adjuvant therapy recommendations generated by an artificial intelligence (AI)-driven tool, TheSerenityBot (TSB), in comparison with Claude-2 and GPT-4, using expert MTB consensus as the reference.

Postoperative breast cancer cases reviewed at the National University Hospital, Singapore, between June and November 2023 were retrospectively analyzed. Eligible patients were women with invasive or preinvasive breast cancer who underwent surgery. Metastatic cases were excluded. TSB, a Claude-2-based model augmented with 2023 National Comprehensive Cancer Network guidelines, generated adjuvant therapy recommendations across seven treatment modalities. Outputs from TSB, Claude-2, and GPT-4 were evaluated for concordance with MTB recommendations. Model performance was assessed using generalized estimating equations.

Fifty patients were included (mean age, 59.8 years); 75.5% had hormone receptor-positive tumors, and 60.0% underwent breast-conserving surgery. TSB demonstrated the highest overall accuracy (0.89), followed by Claude-2 (0.86) and GPT-4 (0.78). GPT-4 showed significantly lower accuracy in genetic testing recommendations (odds ratio [OR], 0.05 [95% CI, 0.015 to 0.149]; P < .001), whereas Claude-2 was less accurate in radiotherapy recommendations (OR, 0.41 [95% CI, 0.17 to 0.98]; P = .040).

A guideline-augmented AI tool such as TSB shows promise in supporting adjuvant therapy decisions in breast cancer. To improve clinical relevance, future iterations will incorporate individualized patient factors, broader guideline frameworks, and electronic health record integration. Prospective trials are ongoing to assess the real-world impact.