Feasibility Study for Using Large Language Models to Identify Goals-of-Care Documentation at Scale in Patients With...

The purpose of our study was to (1) use a large language model (LLM) to identify goals-of-care (GOC) conversations in a large volume of notes, and (2) explore the potential of LLMs for a novel summarization task.

We included patients diagnosed with advanced cancer between April 1, 2024, and June 30, 2024. A validated LLM prompt for GOC was applied to electronic health records (EHRs) using a Health Insurance Portability and Accountability Act (HIPPA)-secure version of GPT-4o, a LLM developed by OpenAI. Output included (1) presence or absence of GOC documentation, (2) explanations with source text used to inform the LLM's determination, and (3) a hallucination score, indicating proportion of source text generated by the LLM that did not perfectly match text in the EHR. Two LLM prompts were designed to generate structured and unstructured GOC summaries. We randomly selected five patients and applied the summarization task to notes flagged by LLM as containing GOC. We reviewed LLM summaries to examine for relevant information.

Among 326 patients associated with nearly 1,400 clinical notes, LLM flagged approximately 40% of notes for GOC documentation. Subsequent review of explanation text identified that 128 patients (nearly 40% of the total patient population) had GOC documentation. The hallucination index for explanations was low, suggesting that the LLM did not produce text that was not found in EHRs. LLM prompts produced accurate summaries in less than 2 minutes per patient.

LLMs can capture GOC at scale and generate clinically useful summaries. Future directions include real-time implementation in the clinical setting.