Clinical reasoning from real-world oncology reports using large language models.

Area of Science:

• Oncology
• Medical Informatics
• Artificial Intelligence

Background:

• Radiology and pathology reports contain critical information for cancer diagnosis and treatment.
• Extracting structured data and performing clinical inferences from these reports is complex and time-consuming.
• Large Language Models (LLMs) show potential for automating these tasks.

Purpose of the Study:

• To evaluate the ability of LLMs to perform structured information extraction and guideline-based clinical inferences from oncology reports.
• To assess the performance of Gemma LLMs with and without structured reasoning prompts.
• To establish a benchmark dataset for evaluating LLM performance in oncology report analysis.

Main Methods:

• A Question Answering (Q&A) benchmark dataset was created from 3650 radiological and 588 pathological reports.
• Tasks included direct extraction of genomic/histological findings and clinical reasoning (RECIST, AJCC TNM staging).
• The Gemma 4B and Gemma 12B models were compared with and without clinical guideline-based structured prompts.

Main Results:

• The Gemma 12B model achieved high F1-scores (92.6-93.3) for direct extraction from pathology reports.
• With structured prompts, Gemma 12B showed significant improvements in reasoning tasks: tumor response (81.5), T-stage (74.3), N-stage (87.1), and M-stage (90.8).
• The Gemma 4B model exhibited inconsistent performance, sometimes degrading with reasoning prompts.

Conclusions:

• LLMs can perform complex guideline-based clinical reasoning on real-world oncology radiology reports.
• Combining clinical criteria (RECIST/AJCC) with structured prompts enables LLMs to support nuanced inference.
• This demonstrates LLMs' potential for future clinical applications in oncology data analysis.