Limitations of Large Language Models in Assisting PI-RADS Scoring on Prostate Biparametric MRI Text Reports

Area of Science:

• Medical imaging analysis
• Artificial intelligence in healthcare
• Prostate cancer diagnostics

Background:

• Prostate cancer (PCa) is a major health concern globally.
• Prostate Imaging Reporting and Data System (PI-RADS) MRI is vital for risk stratification.
• Inter-reader variability in PI-RADS scoring impacts diagnostic consistency, especially in challenging zones and with varying experience levels.
• Large language models (LLMs) offer potential for standardizing medical reports and enhancing diagnostic consistency.

Purpose of the Study:

• To assess LLM performance in assisting Prostate Imaging Reporting and Data System (PI-RADS) scoring from biparametric MRI reports.
• To compare LLM performance against radiologists with diverse experience levels.
• To identify independent predictors of prostate cancer (PCa) and clinically significant PCa (csPCa) using multivariable logistic regression.

Main Methods:

• Retrospective analysis of 210 patients undergoing transperineal biopsy for suspected PCa.
• Independent PI-RADS v2.1 scoring by three radiologists and two LLMs (DeepSeek, ChatGPT-4.1) on anonymized MRI reports.
• Diagnostic performance evaluation using biopsy results, calculating sensitivity, specificity, PPV, NPV, and AUC.
• Subgroup analysis by lesion location (peripheral vs. transition zone) and participant-level analysis with PI-RADS thresholds ≥3 and ≥4.
• Decision curve analysis for clinical utility and multivariable logistic regression for predictor identification.

Main Results:

• Senior radiologist achieved the highest diagnostic performance (AUC 0.847 for PCa, 0.859 for csPCa).
• LLMs demonstrated high sensitivity but significantly lower specificity and positive predictive value (PPV) compared to human readers.
• Senior radiologist outperformed LLMs in the transition zone; LLMs showed high sensitivity but low specificity in the peripheral zone.
• A PI-RADS threshold of ≥4 improved specificity for all readers, with the senior radiologist's scores showing the highest clinical utility.
• PSA density was the strongest independent predictor for PCa and csPCa; only senior radiologist's PI-RADS scores retained independent predictive value.

Conclusions:

• LLMs show high sensitivity for PCa and csPCa detection but lack specificity, particularly in specific MRI regions.
• LLMs are best utilized as adjuncts for indeterminate cases or with higher PI-RADS thresholds (≥4).
• Experienced radiologists demonstrate superior diagnostic performance, emphasizing the need for cautious LLM implementation.
• Future research should focus on enhancing LLM specificity and reliability, potentially through integration with human expertise for improved diagnostic accuracy and efficiency.