Large Language Models as Decision-Making Tools in Oncology: Comparing Artificial Intelligence Suggestions and Expert...

Area of Science:

• Oncology
• Artificial Intelligence
• Medical Informatics

Background:

• Early breast cancer (BC) treatment decisions are complex and benefit from multidisciplinary team meetings (MDTs).
• Large language models (LLMs) are emerging tools with potential applications in healthcare decision support.

Purpose of the Study:

• To evaluate the accuracy of generalist LLMs in recommending appropriate treatment options for early breast cancer patients.
• To compare the performance of different LLMs against expert oncologists' decisions.

Main Methods:

• A retrospective study analyzed anonymized medical records of early BC patients from January to April 2024.
• Three LLMs (Claude3-Opus, GPT4-Turbo, LLaMa3-70B) generated treatment suggestions, compared against expert decisions from MDTs.
• Primary outcome: rate of appropriate LLM suggestions; Secondary outcomes: LLM F1 score and specificity for treatment categories.

Main Results:

• Claude3-Opus (86.6%) and GPT4-Turbo (85.7%) demonstrated high accuracy in treatment suggestions, outperforming LLaMa3-70B (75.0%).
• LLMs excelled in recommending adjuvant endocrine and targeted therapies.
• Overestimation of adjuvant radiotherapy and variable performance in chemotherapy and genomic testing were noted.

Conclusions:

• Claude3-Opus and GPT4-Turbo show significant promise in assisting early BC treatment recommendations.
• LLMs have the potential to enhance MDT decision-making, particularly for adjuvant therapies.
• Further prospective studies and LLM fine-tuning are necessary to confirm clinical utility, especially for surgical validation and genomic testing.