Zero-Shot PI-RADS Version 2.1 Scoring with ChatGPT-4 Turbo and Llama 3: Diagnostic Performance and Agreement with...

This retrospective, single-center study aimed to assess the diagnostic performance and agreement of two large language models (LLMs), ChatGPT-4 Turbo (OpenAI) and Llama 3 (Meta AI), in assigning Prostate Imaging Reporting and Data System (PI-RADS) scores to prostate MRI reports and to compare their performance with two abdominal radiologists. Structured prostate MRI reports (n = 500) obtained between January and December 2022, with original PI-RADS scores removed, were processed with LLMs using a standardized prompt to extract PI-RADS version 2.1 scores. Two abdominal radiologists independently assigned scores, with a third adjudicating discrepancies. Prostate biopsy results served as the reference standard for diagnostic performance assessment. There was high agreement between both models and radiologists: ChatGPT-4 Turbo and Llama 3 achieved 97.7% agreement (κ = 0.95), agreement between the LLM and radiologists ranged from 94.7% to 95.7% (κ = 0.89-0.91), and interradiologist agreement was 94.4% (κ = 0.88). ChatGPT-4 Turbo assigned significantly higher scores than radiologists (P < .005), while differences with Llama 3 were not statistically significant (P = .08). ChatGPT-4 Turbo and the original MRI reports achieved an area under the receiver operating characteristic curve (AUC) of 0.79 for predicting prostate cancer, and radiologists and Llama 3 achieved AUCs of 0.78 each. These results suggest that LLMs could improve prostate MRI reporting through accurate and consistent PI-RADS scoring. Keywords: Prostate, Oncology, Large Language Model Supplemental material is available for this article. © RSNA, 2025.