Radiologist Detection of Deficiencies in LLM-Generated Patient Communications About Radiology Reports: A Multi-Reader...

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Health Informatics

Background:

• Patients increasingly use large language models (LLMs) for medical information.
• The reliability of radiologist oversight in LLM-generated patient communication is unquantified.
• Ensuring accuracy in AI-driven patient interactions is critical for safe healthcare delivery.

Purpose of the Study:

• To evaluate radiologist diagnostic accuracy in detecting LLM response deficiencies.
• To quantify the benefit of multi-reader oversight in identifying errors.
• To assess the safety and reliability of LLM-based communication tools in radiology.

Main Methods:

• Retrospective multi-reader study of 1200 LLM-generated answers to simulated patient questions.
• Answers derived from MIMIC-IV radiology reports and evaluated by three board-certified radiologists.
• Scoring system: 1 (clinically significant error), 2 (incomplete/ambiguous), 3 (accurate); adjudicated consensus as reference standard.

Main Results:

• 9.4% of LLM answers were deficient (1.5% clinically significant errors, 7.9% incomplete/ambiguous).
• Individual radiologists missed 5.6-11.1% of clinically significant errors.
• Dual-reader oversight reduced the average miss rate to 0.6%; triple-reader review detected all deficiencies.

Conclusions:

• Individual radiologist review of LLM responses shows variable sensitivity, with potential for undetected errors.
• Dual-reader oversight significantly enhances the safety and reliability of LLM-based patient communication tools.
• Multi-reader collaboration is essential for mitigating risks associated with AI in patient-facing medical information.