A proof-of-concept study for patient use of open notes with large language models

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Informatics
• Patient Engagement Technologies

Background:

• Large language models (LLMs) are increasingly used by both patients and clinicians.
• While clinician use of LLMs for tasks like managing patient messages is documented, patient utilization for understanding clinical notes remains less explored.
• This study addresses the gap in understanding how patients can leverage LLMs for health information interpretation.

Purpose of the Study:

• To evaluate the reliability and accuracy of commercially available LLMs in responding to patient-generated queries based on an open clinical visit note.
• To compare the performance of different LLMs and prompt strategies in interpreting complex medical information for patients.

Main Methods:

• A cross-sectional proof-of-concept study evaluated three LLMs (ChatGPT 4o, Claude 3 Opus, Gemini 1.5).
• Four prompt series (Standard, Randomized, Persona, Randomized Persona) with patient-designed questions were used against a neuro-oncology progress note.
• Responses were scored by a neuro-oncologist and a patient using an 8-criterion rubric assessing accuracy, relevance, clarity, actionability, empathy, completeness, evidence, and consistency.

Main Results:

• Persona-based prompts, particularly with ChatGPT 4o, yielded the highest scores across all evaluation criteria.
• Standard and Persona prompt series generally performed better than Randomized or Randomized Persona series.
• All evaluated LLMs demonstrated low performance in providing evidence to support their responses.

Conclusions:

• LLMs show significant potential to aid patients in interpreting open clinical notes.
• Employing persona-style prompts is a key strategy for optimizing LLM performance in patient-driven health information queries.
• Further development and patient education are crucial for enhancing LLM utility in patient understanding of health data.