Assessing Large Language Models for Oncology Data Inference From Radiology Reports.

Area of Science:

• Oncology
• Radiology
• Artificial Intelligence

Background:

• Radiology reports contain crucial information for cancer diagnosis and management.
• Large language models (LLMs) are emerging tools for analyzing clinical text.
• Evaluating LLM performance in oncology is essential for clinical integration.

Purpose of the Study:

• To assess the effectiveness of proprietary and open-source LLMs in detecting pancreatic cancer presence, location, and treatment response from radiology reports.
• To compare the performance of various LLMs, including GPT-4, GPT-3.5-turbo, Gemma-7B, and Llama3-8B.

Main Methods:

• Analysis of 203 deidentified pancreatic cancer radiology reports.
• Manual annotation for disease status, location, and indeterminate nodules.
• Utilizing LLMs (GPT-4, GPT-3.5-turbo, Gemma-7B, Llama3-8B) with prompt engineering and ablation strategies.
• Secondary review of discrepancies by an oncologist.

Main Results:

• GPT-4 achieved the highest accuracy (75.5% F1-micro) in inferring disease status.
• Open models Mistral-7B (68.6%) and Llama3-8B (61.4%) showed comparable performance.
• Models generally identified disease locations well, but open models struggled with postsurgical changes.
• GPT-4 and Llama3-8B demonstrated strong precision and recall for disease site identification.

Conclusions:

• LLMs, particularly GPT-4, can effectively extract oncologic insights from radiology reports.
• Summarization strategies enhance LLM performance for clinical support and analytics.
• Open-source LLMs show potential for use in restricted environments.
• The study provides a valuable annotated dataset for future LLM research in oncology.