Performance of ChatGPT (GPT-4o) in automated Lung-Reporting and Data Systems (RADS) classification: a
Vaibhav Gulati1, Donna Parizadeh1, Gabriel Virador1
1Department of Radiology, Trinity Health Mid-Atlantic - Mercy Catholic Medical Center, Darby, PA, USA.
Aim:
The aim of this study is to explore the capabilities of large language models (LLMs), such as ChatGPT, in assigning Lung-Reporting and Data Systems (RADS) scores when given radiology report findings, and to explore prompt engineering practices for efficient and effective integration of artificial intelligence (AI) into radiology workflows.
Material And Methods:
A total of 327 de-identified low-dose lung screening computed tomography (CT) reports were evaluated using GPT- 4o, with instructions to assign Lung-RADS scores based on the reports provided. Structured and unstructured zero-shot prompting strategies were tested. Accuracy, overcalls, undercalls, and weighted Cohen's kappa (linear weights) were computed for each prompt type compared with the rater. McNemar's test was used to compare accuracy between structured and unstructured prompts.
Results:
Agreement between GPT outputs and the rater's reference standard was higher for the structured prompt compared with the unstructured prompt. For the structured prompt, overall accuracy was 68.8% with a weighted Cohen's kappa of 0.758, while the unstructured prompt achieved an accuracy of 58.1% and a weighted Cohen's kappa of 0.695. Structured prompting produced fewer overcalls and more undercalls compared with unstructured prompting.
Conclusion:
LLMs demonstrate promise in automatically assigning scores to screening imaging studies when provided with findings and can potentially be incorporated into reporting workflows to improve reporting efficiency.
Related Concept Videos
01:25Pulmonary Function Tests
Pulmonary Function Tests are crucial diagnostic tools for assessing respiratory function, particularly in patients with chronic respiratory disorders. They comprehensively evaluate lung volumes, ventilatory function, breathing mechanics, diffusion, and gas exchange. These tests help diagnose pulmonary diseases and play a significant role in monitoring disease progression, evaluating disability, and assessing response to therapy.
PFTs involve using a spirometer, a...
01:13Radiological Investigation III: Pulmonary Angiogram and PET Scan
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...