Leveraging large language models for generating responses to patient messages-a subjective analysis

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Clinical Communication

Background:

• Patient portal messaging is crucial for healthcare communication.
• Current response generation methods may lack efficiency and personalization.
• Large language models (LLMs) offer potential for automated response generation.

Purpose of the Study:

• To develop and evaluate fine-tuned LLMs for generating responses to patient portal messages.
• To assess the performance of these models in terms of empathy, responsiveness, accuracy, and usefulness.

Main Methods:

• Developed CLAIR-Short and CLAIR-Long models using a pre-trained LLM (LLaMA-65B) and a large dataset of patient messages and responses.
• Augmented responses with patient education, empathy, and professionalism using OpenAI API.
• Evaluated model performance by having primary care physicians rate responses from CLAIR models and ChatGPT.

Main Results:

• CLAIR-Short generated concise responses, while CLAIR-Long provided enhanced educational content.
• CLAIR-Long responses were rated similarly to ChatGPT, with positive scores for responsiveness, empathy, and accuracy.
• Physicians found CLAIR-Long responses to be neutrally useful.

Conclusions:

• Fine-tuned LLMs hold significant potential for improving patient-provider communication via electronic health record portals.
• LLM-generated responses can enhance patient education and streamline communication.
• Further research is needed to optimize usefulness and clinical integration.