Open LLM-based actionable incidental finding extraction from [¹⁸F]fluorodeoxyglucose PET-CT radiology reports

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• [18F]fluorodeoxyglucose positron emission tomography-computed tomography ([18F]FDG PET-CT) reports frequently contain actionable incidental findings (AIFs) that impact patient management.
• Automated extraction of AIFs from these reports is crucial for timely clinical intervention.
• Current methods for AIF identification are often manual and time-consuming.

Purpose of the Study:

• To develop and evaluate an open, large language model (LLM)-based pipeline for extracting AIFs from [18F]FDG PET-CT reports.
• To enable both short-term clinical applications (e.g., alerts) and long-term research using structured AIF data.
• To assess the performance of the LLM pipeline on both internal and external datasets.

Main Methods:

• A pipeline was created using an LLM fine-tuned with QLoRA and chain-of-thought (CoT) prompting on annotated [18F]FDG PET-CT reports.
• The pipeline classifies reports for AIFs, extracts relevant sentences, and stores findings in JSON format.
• Performance was evaluated using F1 scores on internal and external test datasets, with both quantitative and qualitative analyses.

Main Results:

• The pipeline achieved high document-level F1 scores (0.917 on internal, 0.79 on external test datasets).
• Sentence-level F1 scores were 0.754 (internal) and 0.522 (external), with qualitative analysis indicating practical utility exceeding quantitative scores.
• Llama-3.1-8B Instruct, adapted with QLoRA and CoT prompting, demonstrated optimal performance and efficiency.

Conclusions:

• A QLoRA-adapted LLM with CoT prompting can reliably extract AIF information from PET-CT reports.
• The developed pipeline offers significant potential for clinical decision support, including alerts and reminders.
• The model's effectiveness in extracting AIFs can aid in managing patient comorbidities and improving healthcare outcomes.