Radiologist Detection of Deficiencies in LLM-Generated Patient Communications About Radiology Reports: A Multi-Reader...

Patients increasingly use large language models (LLMs) to interpret radiology reports, yet the reliability of radiologist oversight in detecting errors in patient-facing LLM responses remains unquantified. This study evaluated the diagnostic accuracy of radiologists in detecting deficiencies in LLM-generated answers to simulated patient questions and quantified the incremental benefit of multi-reader oversight. In this retrospective multi-reader study, three board-certified radiologists independently evaluated 1200 LLM-generated answers (from three different models; 400 per model) to questions derived from 187 MIMIC-IV radiology reports. Answers were scored on a 3-point scale: 1 (clinically significant error), 2 (incomplete/ambiguous), and 3 (accurate). An adjudicated consensus served as the reference standard. Individual reader sensitivity, specificity, and multi-reader miss rates were compared. Of 1200 answers, 113 (9.4%) were deficient by consensus, including 18 clinically significant errors (1.5%) and 95 incomplete/ambiguous responses (7.9%). For clinically significant errors, readers missed 5.6-11.1% of cases. Across all deficiencies, individual reader sensitivity ranged from 81.4% (95% CI, 73.3-87.5) to 96.5% (95% CI, 91.3-98.6), with significant differences (McNemar p < 0.001). A single reader missed an average of 12.4% of deficiencies, whereas dual-reader review reduced the average miss rate to 0.6%; triple-reader review detected all deficiencies. Individual radiologist review of LLM-generated patient communications showed variable sensitivity, with some clinically significant errors remaining undetected. Dual-reader oversight significantly improves the safety and reliability of LLM-based communication tools.