The Importance of Primary Care Subject Matter Experts: Output Quality in Large Language Models Prompt Engineering.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Primary Care Technology

Background:

• Large language models (LLMs) present opportunities for primary care (PC) by automating documentation, summarizing patient records, and aiding communication.
• The clinical utility of LLMs is contingent upon their outputs aligning with practical, real-world healthcare requirements.

Purpose of the Study:

• To describe the development of Primary Care (PC) Navigator, a multimodal AI tool integrating clinical encounter data with LLMs.
• To highlight the significance of prompt engineering in optimizing LLM performance for clinical applications.

Main Methods:

• Development of a multimodal AI tool (PC Navigator) combining audio/video from clinical encounters with an LLM.
• Pilot study focused on prompt design and iterative refinement through clinician-engineer collaboration.
• Utilized the CARE (Context, Action, Result, Example) framework for prompt engineering.

Main Results:

• Prompt design critically influences the quality and clinical relevance of LLM outputs.
• Collaborative prompt engineering led to more effective AI-generated behavior change plans.
• The CARE framework facilitates the creation of accurate, relevant, and actionable LLM outputs.

Conclusions:

• Clinician involvement is vital for developing effective LLMs in primary care, shifting them from users to co-developers.
• Effective prompt engineering ensures AI tools are technically sound and aligned with patient needs and clinical practice realities.
• Multimodal AI tools, guided by clinician input, can significantly support primary care functions.