Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation

Plain language summaries (PLSs) are essential for facilitating effective communication between clinicians and patients by making complex medical information easier for laypeople to understand and act upon. Large language models (LLMs) have recently shown promise in automating PLS generation, but their effectiveness in supporting health information comprehension remains unclear. Prior evaluations have generally relied on automated scores that do not measure understandability directly, or subjective ratings from convenience samples with limited generalizability. To address these gaps, we conducted a large-scale crowdsourced evaluation of LLM-generated PLSs using Amazon Mechanical Turk with 150 participants. We assessed PLS quality through subjective perceived ratings of simplicity, informativeness, coherence, and faithfulness; and task-based measures of reader performance, including multiple-choice accuracy as an operational proxy for comprehension and recall as a complementary measure of gist-level retention. Additionally, we examined the alignment between 10 automated evaluation metrics and human judgments. Our results show that participants often rated LLM-generated PLSs as similarly clear and coherent as human-authored summaries, but participants performed significantly better on the comprehension questions after reading human-written PLSs. This divergence between perceived quality and actual understanding suggests that fluent, trustworthy-sounding AI-generated summaries may engender confidence without reliably supporting comprehension. Furthermore, automated evaluation metrics fail to reflect human judgment, calling into question their suitability for evaluating PLSs. This is the first study to systematically evaluate LLM-generated PLSs based on both reader preferences and comprehension outcomes. Our findings highlight the need for evaluation frameworks that move beyond surface-level quality and for generation methods that explicitly optimize for layperson comprehension, as fluent AI-generated summaries may make readers feel confident without truly understanding the underlying health information.