Evaluating Large Language Model-Generated Clinical Summaries Through a Dual-Perspective Framework: Retrospective...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Patient Communication

Background:

• Large language models (LLMs) are emerging tools for patients and families to understand complex medical information.
• Current evaluations of LLMs in healthcare primarily focus on clinician-assessed accuracy, potentially overlooking patient perspectives.
• Interpreting pediatric cardiac intensive care unit (CICU) documentation presents unique challenges due to its complexity.

Purpose of the Study:

• To evaluate the performance of GPT-4o mini in summarizing pediatric CICU notes.
• To compare parent and physician assessments of LLM-generated summaries regarding readability, clinical fidelity, and helpfulness.
• To identify discrepancies between patient and clinician viewpoints on the utility of LLM summaries.

Main Methods:

• Fifty pediatric CICU notes were summarized using the GPT-4o mini large language model.
• Physicians and parents independently reviewed the generated summaries.
• Participants rated the summaries based on readability, clinical fidelity, and overall helpfulness.

Main Results:

• Significant differences were observed between parent and physician ratings, particularly concerning the helpfulness of the summaries.
• Clinicians provided key insights into the clinical accuracy of the LLM-generated content.
• Parents offered valuable feedback on the readability and understandability of the summaries.

Conclusions:

• Evaluating LLMs in healthcare requires a dual-perspective framework incorporating both clinician and patient assessments.
• Balancing clinical precision with patient understanding is crucial for effective LLM implementation in patient-facing medical documentation.
• Future research should focus on developing and validating such comprehensive evaluation methodologies.