Leveraging Large Language Models for Decision Support in Personalized Oncology

Area of Science:

• Oncology
• Biomedical Informatics
• Artificial Intelligence

Background:

• Precision oncology relies on manual interpretation of complex biomarkers.
• Large language models (LLMs) offer potential for automating clinical decision support.

Purpose of the Study:

• To evaluate the performance of four LLMs as support tools in precision oncology.
• To define the role of LLMs in identifying personalized cancer treatment options.

Main Methods:

• A diagnostic study involving 10 fictional advanced cancer cases with genetic alterations.
• Four LLMs (ChatGPT, Galactica, Perplexity, BioMedLM) and an expert physician generated treatment options.
• Molecular tumor boards assessed LLM-generated options for recognizability and clinical usefulness.

Main Results:

• LLMs generated more treatment options than human experts but with lower precision and recall (F1 scores 0.04-0.19).
• Combined LLM output improved performance (F1 score 0.29).
• LLM options were identifiable as AI-generated, often due to lacking evidence, yet at least one LLM option was helpful per case, with some unique useful options identified solely by LLMs.

Conclusions:

• LLM-generated treatment options in precision oncology currently lack the quality and credibility of human experts.
• LLMs can provide helpful, complementary ideas and assist in literature screening for evidence-based personalized cancer treatment.