Simplifying radiology reports with large language models: privacy-compliant open- versus closed-weight models

Area of Science:

• Artificial Intelligence in Medical Imaging
• Natural Language Processing in Healthcare
• Radiology Report Simplification

Background:

• Large language models (LLMs) offer potential for simplifying complex medical information.
• Privacy concerns surrounding closed-weight LLMs limit their direct clinical application in healthcare.
• Developing privacy-compliant methods for generating patient-friendly radiology reports is crucial.

Purpose of the Study:

• To compare the effectiveness of closed-weight and open-weight LLMs in creating patient-friendly radiology reports.
• To evaluate the readability and understandability of LLM-generated reports for medical laypersons.
• To assess the clinical applicability of in-hospital deployed, privacy-compliant open-weight LLMs.

Main Methods:

• Sixty radiology reports (X-ray, ultrasound, CT, MRI) were translated using GPT-4o (closed-weight) and Llama-3-70b, Mixtral-8x22B (open-weight).
• Readability was assessed using Flesch reading ease, reading time, and word/sentence counts.
• Understandability was rated by 21 medical laypeople on a 5-point Likert scale; statistical analysis employed linear mixed-effects models and H-Kruskal-Wallis tests.

Main Results:

• LLM-generated reports showed significantly improved readability (higher Flesch scores) compared to original reports (p < 0.001).
• Layperson understandability ratings were markedly higher for all LLM reports (p < 0.001), with GPT-4o and Llama-3-70B performing comparably (p=0.136).
• Open-weight models (Mixtral-8x22B, Llama-3-70B) had more potential harm errors than GPT-4o (p=0.005, p=0.025).

Conclusions:

• LLMs substantially enhance layperson comprehension of radiology reports.
• Privacy-compliant, open-weight LLMs demonstrate significant potential for clinical integration.
• Human oversight is still necessary when using LLMs for generating patient-facing medical reports.