Privacy-Preserving Large Language Model for Matching Findings and Tracking Interval Changes in Longitudinal Radiology...

In current radiology practice, radiologists identify a finding in the current imaging exam, manually match it against the description from the prior exam report and assess interval changes. Large Language Models (LLMs) can identify report findings, but their ability to track interval changes has not been tested. The goal of this study was to determine the utility of a privacy-preserving LLM for matching findings between two reports (prior and follow-up) and tracking interval changes in size. In this retrospective study, body MRI reports from NIH (internal) were collected. A two-stage framework was employed for matching findings and tracking interval changes. In Stage 1, the LLM took a sentence from the follow-up report and discovered a matched finding in the prior report. In Stage 2, the LLM predicted the interval change status (increase, decrease, or stable) of the matched findings. Seven LLMs were locally evaluated and the best LLM was validated on an external non-contrast chest CT dataset. Agreement with the reference (radiologist) was measured using Cohen's Kappa (κ). The internal body MRI dataset had 240 studies (120 patients, mean age, 47 ± 16 years; 65 men) and the external non-contrast chest CT dataset contained 134 studies (67 patients, mean age, 58 ± 18 years; 44 men). On the internal dataset, TenyxChat-7B LLM fared the best for matching findings with an F1-score of 85.4% (95% CI: 80.8, 89.9) over the other LLMs (p < 0.05). For interval change detection, the same LLM achieved a 62.7% F1-score and showed a moderate agreement (κ = 0.46, 95% CI: 0.37, 0.55). For the external dataset, the same LLM attained F1-scores of 81.8% (95% CI: 74.4, 89.1) for matching findings and 77.4% for interval change detection respectively, with a substantial agreement (κ = 0.64, 95% CI: 0.49, 0.80). The TenyxChat-7B LLM used for matching longitudinal report findings and tracking interval changes showed moderate to substantial agreement with the reference standard. For structured reporting, the LLM can pre-fill the "Findings" section of the next follow-up exam report with a summary of longitudinal changes to important findings. It can also enhance the communication between the referring physician and radiologist.