Prompt engineering on leveraging large language models in generating response to InBasket messages

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Communication

Background:

• High volumes of patient medical advice requests (PMARs) present a significant challenge in healthcare delivery.
• Large Language Models (LLMs) are being explored as a potential solution to streamline communication and reduce clinician workload.

Purpose of the Study:

• To investigate the efficacy of prompt engineering in developing LLM-generated draft responses for PMARs.
• To assess the quality and satisfaction of both patients and clinicians with LLM-generated responses.

Main Methods:

• A novel human-involved iterative process was used to train and validate LLM prompts for PMAR response generation.
• GPT-4 was utilized, with prompts refined based on clinician and patient feedback across multiple iterations.
• The optimized prompt was validated on independent datasets and tested in a real-world electronic health record production environment with primary care clinicians.

Main Results:

• Physician acceptance of draft suitability significantly increased from 62% to 84% after prompt engineering.
• Patient favorability of message tone (78%) and overall quality (80%) improved.
• Patients could not differentiate between human and LLM-generated responses 76% of the time, and 72% of clinicians felt LLMs could reduce cognitive load.

Conclusions:

• Iterative prompt engineering, informed by synergistic clinician and patient feedback, is effective in generating clinically relevant and useful draft responses to PMARs.
• LLM-driven draft responses have the potential to enhance efficiency and satisfaction in managing patient communications.