DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Discharge

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Informatics
• Patient Education

Background:

• Discharge communication is crucial for patient care transitions, shifting focus to education.
• Existing benchmarks for large language models (LLMs) primarily assess diagnostic reasoning, neglecting post-visit patient support.
• Effective patient education post-discharge remains an underexplored area for LLM evaluation.

Purpose of the Study:

• To introduce DischargeSim, a novel benchmark for evaluating LLMs' capabilities as personalized discharge educators.
• To assess LLMs' performance in simulating realistic, multi-turn patient-doctor conversations post-discharge.
• To measure LLM effectiveness across dialogue quality, personalized document generation, and patient comprehension.

Main Methods:

• DischargeSim simulates post-discharge conversations between LLM-driven DoctorAgents and PatientAgents with varied psychosocial backgrounds.
• Conversations cover six clinical discharge topics, evaluated on dialogue quality, document generation (summaries, checklists), and patient understanding via exams.
• Experiments involved 18 LLMs interacting with diverse simulated patient profiles.

Main Results:

• Significant performance gaps were observed in LLMs' discharge education abilities.
• LLM performance varied considerably based on patient psychosocial profiles.
• Larger model size did not consistently correlate with improved educational outcomes, indicating strategic trade-offs.

Conclusions:

• DischargeSim provides a foundational benchmark for assessing LLMs in post-visit clinical education.
• Current LLMs exhibit limitations in delivering equitable and personalized patient discharge support.
• Further development is needed to enhance LLM capabilities for effective patient education and communication post-discharge.