Improved Readability and Translational Instability in LLM-Generated Radiology Reports

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Radiology Reporting

Background:

• Large language models (LLMs) offer potential for translating complex radiology reports into patient-friendly language.
• However, the inherent instability of LLM outputs poses challenges for clinical integration.

Purpose of the Study:

• To quantitatively evaluate the translational accuracy, error rates, and instability of LLMs in generating patient-centric radiology reports.
• To assess the impact of demographic factors on the readability of LLM-generated reports.

Main Methods:

• A retrospective analysis of 320 de-identified radiology reports processed by three LLMs.
• Utilized a two-stage prompt engineering strategy (baseline and optimized).
• Evaluated by senior radiologists for medical accuracy and by non-medical participants for readability, stratified by demographics.

Main Results:

• All LLMs demonstrated instability, information omission, and generalized recommendations.
• Optimized structured prompts significantly reduced output variance and improved translational accuracy (notably DeepSeek-R1 and ChatGPT-4.0).
• LLMs enhanced report readability (P < 0.05), with DeepSeek-R1 performing best; demographic factors influenced patient comprehension.

Conclusions:

• LLMs can improve radiology report readability but exhibit inherent instability and information omission.
• Optimized prompting strategies can mitigate variability and enhance translation accuracy.
• LLMs are best utilized as human-supervised auxiliary tools, not standalone solutions.