Comparative Evaluation of Large Language Models in Explaining Radiology Reports: Expert Assessment of Readability,...

Area of Science:

• Artificial Intelligence in Radiology
• Natural Language Processing in Healthcare
• Medical Communication

Background:

• Radiology reports contain complex medical information.
• Patients often struggle to understand their radiology reports.
• Large language models (LLMs) offer potential for improving patient comprehension.

Purpose of the Study:

• To compare the understandability, readability, and communication characteristics of radiology report explanations generated by ChatGPT, Gemini, and Copilot.
• To assess the medical correctness, patient guidance, and anxiety-inducing potential of LLM-generated explanations.
• To inform the optimal use of AI in radiological communication.

Main Methods:

• 100 anonymized radiology reports from five subspecialties were analyzed.
• Reports were explained by ChatGPT (GPT-3.5), Gemini, and Copilot using a standardized prompt.
• Responses were evaluated for medical correctness, understandability (PEMAT-U), readability (FRE, ARI, GFI), and communication features.

Main Results:

• All models achieved high medical correctness (mean 1.97/2).
• ChatGPT yielded the most readable and understandable explanations (p < 0.01).
• Gemini excelled in patient guidance, Copilot in uncertainty language and clinical suggestions.

Conclusions:

• ChatGPT demonstrated superior readability and understandability for radiology report explanations.
• Differences in AI model outputs support context-specific application for enhanced patient communication.
• Findings guide the strategic implementation of AI tools to improve patient engagement with radiological findings.