Simulation-Based Evaluation of a Large Language Model-Enabled Clinical Decision Support Platform in Oncology

Area of Science:

• Oncology
• Clinical Informatics
• Artificial Intelligence in Medicine

Background:

• Electronic health record (EHR) inefficiencies contribute to clinician cognitive overload and burnout.
• Synthesizing fragmented patient data into coherent summaries is a critical but time-consuming clinical task.

Purpose of the Study:

• To evaluate a large language model (LLM)-enabled clinical decision support (LLM-CDS) platform against a simulated EHR (SimEPR).
• To assess the impact on workflow efficiency and user experience in generating clinical summaries for tumor board preparation.
• To explore LLM-CDS applicability in other oncology workflows like consultation preparation and treatment planning.

Main Methods:

• A within-participant simulation involving 26 oncologists using synthetic breast cancer cases.
• Comparison of LLM-CDS (editable LLM-generated summaries) with SimEPR (manual composition) for task completion time.
• Assessment of summary quality (completeness, correctness, conciseness) and user experience (usability, cognitive load, acceptability).

Main Results:

• LLM-CDS significantly reduced summary completion time (6:55 vs 8:47 minutes; P < .001).
• Summary completeness was higher with LLM-CDS (3.93 vs 3.13), while correctness and conciseness were similar.
• 87% of participants would recommend LLM-CDS, citing anticipated time savings and usefulness (92%) for tumor board and consultation preparation.

Conclusions:

• The LLM-CDS platform enhances the efficiency and completeness of clinical summarization.
• Strong user acceptance and perceived time savings indicate potential for streamlining various oncology workflows.
• LLM-CDS demonstrates promise in mitigating clinician burden associated with EHR data synthesis.