Generating patient-friendly radiology reports with a local large language model: expert evaluation of quality and readability

To assess the feasibility of using a locally hosted large language model (LLM) to generate patient-friendly radiology reports while avoiding cloud transfer of report text.

In this retrospective study, 360 chest, knee, lumbar spine, and shoulder X-rays, and shoulder and abdominal ultrasound reports from Dutch primary care were included. Main report sections (n = 359) and conclusion sections (n = 350) were translated into patient-friendly versions using Gemma-3 (12B parameters) and assessed for accuracy and completeness on 5-point Likert scales. Readability was evaluated using text length and Flesch-Douma reading ease (FDRE) scores. Hallucinations were systematically documented. Statistical analyses included Kruskal-Wallis, Wilcoxon signed-rank, and paired t-tests.

Patient-friendly conclusions achieved higher median accuracy scores than main reports (5 [Q1-Q3 5-5] vs 4 [3-5]), while completeness scores were high for both sections (5 [5-5]). Compared with original reports, patient-friendly reports were longer (median increase 65 words [48-84], p < 0.001) and had higher FDRE scores (mean increase 54.2 ± 18.8, p < 0.001). Overall, 71.0% of main reports and 96.0% of conclusions met predefined quality criteria (accuracy and completeness ≥ 4), while 48.0% of combined main report and conclusion sections met both quality and readability criteria (FDRE score > 60). Clinically relevant hallucinations occurred in 45 reports (12.5%).

A locally hosted LLM demonstrated feasibility for generating patient-friendly translations of Dutch X-ray and ultrasound reports while avoiding cloud transfer of data, with better performance for conclusion sections than main reports. However, clinically relevant hallucinations remain a safety concern and indicate the need for clinician oversight and further evaluation before clinical implementation.

Question Can a locally hosted LLM generate patient-friendly radiology reports while preserving data privacy and maintaining acceptable accuracy, completeness, and readability? Findings A locally hosted LLM generated patient-friendly report conclusions with higher quality scores than main reports, while clinically relevant hallucinations remained an important safety concern. Clinical relevance A locally hosted LLM has potential to generate patient-friendly radiology report translations without external cloud data transfer. Report conclusions appear to be the most promising initial application, although clinician oversight and further patient-centered validation remain essential before clinical implementation.