Automated Resectability Classification of Pancreatic Cancer CT Reports with Privacy-Preserving Open-Weight Large...

Area of Science:

• Artificial Intelligence in Radiology
• Natural Language Processing for Medical Imaging
• Oncology Data Extraction

Background:

• Radiology reports contain crucial information for pancreatic ductal adenocarcinoma (PDAC) treatment planning.
• Extracting key features and determining resectability status from these reports is vital for patient management.
• Manual extraction is time-consuming and prone to variability.

Purpose of the Study:

• To assess the efficacy of open-weight large language models (LLMs) in extracting radiological features and NCCN resectability status from PDAC reports.
• To compare the performance of Gemma-2-27b-it and Llama-3-70b-instruct models on this task.
• To evaluate model performance on both internal and external datasets.

Main Methods:

• Developed and validated prompts using fictitious and real PDAC radiology reports.
• Established ground truth for 18 key features and resectability by two radiologists.
• Evaluated Gemma-2-27b-it and Llama-3-70b-instruct using recall, precision, F1-score, extraction accuracy, and resectability accuracy.
• Performed statistical analyses including McNemar's test and mixed-effects logistic regression.

Main Results:

• Llama-3-70b-instruct showed higher internal validation recall (99% vs. 95%) and extraction accuracy (98% vs. 97%) than Gemma-2-27b-it.
• Both models achieved 96% recall and extraction accuracy on the internal test set.
• On the external test set, recall was 93% for both models, with Gemma achieving slightly higher extraction accuracy (95% vs. 93%).
• Overall resectability accuracy was higher internally for Llama (95%) than Gemma (93%), and externally for Gemma (89%) than Llama (83%), though not statistically significant.

Conclusions:

• Open-weight LLMs can accurately extract PDAC radiological features and determine NCCN resectability status.
• Model performance is robust on internal data but may decrease on external datasets.
• Institution-specific optimization is necessary to maximize the utility of LLMs in clinical practice.