Leveraging Fine-Tuned Large Language Models for Interpretable Pancreatic Cystic Lesion Feature Extraction and Risk...

BACKGROUND. Manual extraction of pancreatic cystic lesion (PCL) features from radiology reports is labor-intensive, limiting large-scale studies needed to advance PCL research. OBJECTIVE. The purpose of this study was to evaluate GPT-4o (closed source [ OpenAI]), Llama (Llama-3.1-8B-Instruct, open source), and DeepSeek (DeepSeek-R1-Distill-Llama-8B, open source) large language models (LLMs) for PCL feature extraction, without and with chain-of-thought (CoT) reasoning. METHODS. We curated a dataset of 6469 abdominal MRI or CT reports (2005-2024) that described PCLs from 5615 patients. Llama and DeepSeek were fine-tuned using quantized low-rank adaptation on GPT-4o-generated CoT labels for extracting PCL and main pancreatic duct features. Features were mapped to risk categories per institutional policy. Evaluation was performed on 285 held-out human-annotated reports from 281 patients. Model outputs for 100 cases were independently reviewed by three radiologists. Feature extraction was evaluated using exact-match accuracy, risk categorization with a macro-averaged F1 score, and radiologist-model agreement with Fleiss kappa values. Error analyses were performed to assess how and why models made mistakes. RESULTS. CoT fine-tuned LLMs showed a feature extraction accuracy of 97% (95% CI, 97-98%) for Llama, 98% (95% CI, 97-98%) for DeepSeek, and 97% (95% CI, 97-98%) for GPT-4o. Risk categorization F1 scores were 0.93 (95% CI, 0.89-0.97) for Llama, 0.94 (95% CI, 0.90-0.98) for DeepSeek, and 0.97 (95% CI, 0.93-0.99) for GPT-4o. Radiologist interreader agreement was high (κ = 0.888) and showed no significant difference with the addition of Llama (κ = 0.882; p > .99), DeepSeek (κ = 0.893, p > .99), or GPT-4o (κ = 0.897, p > .99). Across all models, object identification and clinical reasoning were the most frequent error types, accounting for 29.3-37.3% and 18.1-21.1% of total errors, respectively. CONCLUSION. LLMs show feasibility for automatically extracting PCL features from radiology reports. Fine-tuned open-source LLMs achieved performance comparable to that of GPT-4o. CoT reasoning improved accuracy and enabled interpretable error analysis. Model-assigned risk categories showed high agreement with abdominal radiologists. CLINICAL IMPACT. LLMs have the potential to enable creation of large structured registries from existing radiology reports to support population-level research on PCLs.