Leveraging Fine-Tuned Large Language Models for Interpretable Pancreatic Cystic Lesion Feature Extraction and Risk...

Area of Science:

• Artificial Intelligence in Radiology
• Medical Informatics
• Natural Language Processing

Background:

• Manual extraction of pancreatic cystic lesion (PCL) features from radiology reports is time-consuming, hindering large-scale research.
• Automating this process is crucial for advancing PCL studies.

Purpose of the Study:

• To evaluate the performance of GPT-4o, Llama, and DeepSeek large language models (LLMs) for extracting PCL features.
• To assess the impact of chain-of-thought (CoT) reasoning on LLM performance for PCL feature extraction.

Main Methods:

• A dataset of 6469 abdominal MRI/CT reports was curated.
• Open-source LLMs (Llama, DeepSeek) were fine-tuned using CoT labels generated by GPT-4o.
• Performance was evaluated using exact match accuracy, risk categorization F1 scores, and radiologist agreement (Fleiss' kappa).

Main Results:

• Fine-tuned LLMs achieved high feature extraction accuracy (97-98%) and risk categorization F1 scores (0.93-0.97).
• Open-source LLMs performed comparably to GPT-4o, with no significant difference in radiologist agreement.
• Object identification and clinical reasoning were identified as primary error sources.

Conclusions:

• LLMs demonstrate feasibility for automated PCL feature extraction from radiology reports.
• Fine-tuned open-source LLMs offer a viable alternative to closed-source models.
• CoT reasoning enhances accuracy and facilitates interpretable error analysis, supporting population-level PCL research.