Comparing large language models and human experts in interpreting MRI reports for personalized patient education

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Patient Education

Background:

• Knee osteoarthritis (OA) is a widespread condition impacting global health.
• Magnetic Resonance Imaging (MRI) is crucial for diagnosing knee OA but technical reports are difficult for patients to understand.
• Personalized patient education derived from MRI reports is challenging to produce efficiently and effectively.

Purpose of the Study:

• To compare the effectiveness of advanced large language models (LLMs) against experienced clinicians in generating patient education materials from knee MRI reports.
• To evaluate LLM-generated content for comprehensibility, personalization, and efficiency.

Main Methods:

• A comparative study involving two LLMs (GPT-4o, Claude 3.5 Sonnet) and experienced clinicians.
• Generation of personalized patient education materials from 150 anonymized knee MRI reports.
• Evaluation using a framework assessing readability, content personalization, and generation efficiency (words per minute).

Main Results:

• Both LLMs significantly outperformed clinicians in understandability, personalization, and efficiency.
• GPT-4o demonstrated superior performance over Claude 3.5 Sonnet and clinicians.
• LLM-generated content showed higher expert-rated understandability, better personalization scores, and markedly higher generation efficiency.

Conclusions:

• Advanced LLMs, especially GPT-4o, excel at translating knee MRI reports into understandable and personalized patient education.
• LLMs offer advantages in readability, personalization, and efficiency over clinician-generated materials.
• LLMs show potential as clinician-supervised tools for scalable patient education, requiring further validation.