Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation.

Area of Science:

• Health Communication
• Artificial Intelligence in Medicine
• Patient Education

Background:

• Plain language summaries (PLSs) are crucial for patient understanding of medical information.
• Large language models (LLMs) show potential for automating PLS generation.
• Previous evaluations of LLM-generated PLSs lack robust measures of comprehension and generalizability.

Purpose of the Study:

• To conduct a large-scale crowdsourced evaluation of LLM-generated PLSs.
• To assess both subjective quality and objective reader performance (comprehension and recall).
• To examine the alignment between automated metrics and human judgments of PLS quality.

Main Methods:

• A crowdsourced study involving 150 participants via Amazon Mechanical Turk.
• Assessment of PLS quality via perceived ratings (simplicity, informativeness, coherence, faithfulness).
• Task-based measures including multiple-choice accuracy for comprehension and recall tests.

Main Results:

• Participants rated LLM-generated PLSs similarly to human-written ones in clarity and coherence.
• However, participants demonstrated significantly better comprehension after reading human-written PLSs.
• Automated evaluation metrics showed poor alignment with human judgments of PLS quality.

Conclusions:

• LLM-generated PLSs may appear fluent and trustworthy but do not reliably enhance patient comprehension.
• Current automated metrics are inadequate for evaluating the true effectiveness of PLSs.
• Future research must focus on developing generation methods and evaluation frameworks that prioritize layperson comprehension.