Automated report-based tracking of pancreatic cysts: implications for guideline-defined growth classification

To develop and evaluate a hybrid large language model (LLM)- and rule-based algorithm for tracking pancreatic cystic lesions (PCLs) across serial radiology reports and apply it to a single-center cohort to characterize cyst growth using first-to-last and consecutive comparisons.

This retrospective study included all abdominal imaging reports interpreted by abdominal radiologists between 2005 and 2024. A validated LLM identified reports describing PCLs. A hybrid algorithm combining LLM-extracted cyst features (size, location) with rule-based logic matched dominant cysts across serial reports. Two blinded radiologists independently annotated cyst matches in a subset of reports. Agreement between algorithm and radiologists was assessed using percent agreement and Cohen's κ. Discrepancies were adjudicated by a third radiologist for error analysis. Absolute size-based growth was assessed according to American College of Radiology (ACR) criteria, whereas cyst growth rate as a worrisome feature was assessed per Kyoto criterion, using both first-to-last and consecutive studies. Predictors of meeting the Kyoto cyst growth rate criterion were analyzed using multivariable Cox regression.

In 58 patients with 418 longitudinal reports, agreement with the adjudicated reference standard was 96%. Cohen's κ was 0.81 (95% CI: 0.66-0.91). The overall cohort included 8,176 patients (61.8% female; mean age 67.6 ± 11.7 years) with 8,389 cysts across 28,489 reports over 40.6 ± 28.5 months (annual growth rate = 0.35 mm/year). ACR absolute growth criteria classified a higher proportion of cysts than the Kyoto cyst growth rate criterion using first-to-last examinations (16.3% vs. 7.4%, p < 0.001), whereas Kyoto cyst growth rate criterion identified a higher proportion using consecutive examinations (21.4% vs. 14%, p < 0.001). Larger baseline cyst size predicted meeting the Kyoto cyst growth rate criterion (HR 1.9-4.0; highest HR 4.0, 95% CI 3.08-5.21; p < 0.001).

A hybrid LLM-rule-based algorithm enables reliable automated longitudinal tracking of PCLs across serial radiology reports. When applied at scale, cyst growth classification varied according to the guideline framework and interval comparison method used, and baseline cyst size was the strongest predictor of meeting Kyoto growth rate threshold.