Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs

Area of Science:

• Clinical Medicine
• Artificial Intelligence
• Medical Informatics

Background:

• Large language models (LLMs) show promise in clinical medicine.
• Effective knowledge transfer from computer science to clinical applications is essential.
• Prompt engineering is a key method for optimizing LLM performance.

Purpose of the Study:

• To evaluate the impact of prompt engineering on LLM reliability and accuracy in clinical medicine.
• To assess LLM agreement with the American Academy of Orthopedic Surgeons (AAOS) osteoarthritis (OA) guidelines.
• To compare the consistency of different prompts and LLMs.

Main Methods:

• Designed and applied various prompt styles to different LLMs.
• Queried LLMs regarding adherence to AAOS OA evidence-based guidelines.
• Repeated each query five times to assess reliability.
• Analyzed consistency across different evidence levels and prompt types.

Main Results:

• GPT-4-Web with ROT prompting achieved the highest overall consistency (62.9%).
• This prompt style demonstrated strong performance for significant recommendations (77.5% consistency).
• LLM reliability varied significantly across prompts and models (Fleiss kappa: -0.002 to 0.984).

Conclusions:

• Prompt engineering significantly influences LLM performance in clinical contexts.
• The ROT prompt for GPT-4-Web emerged as the most consistent method.
• Careful prompt selection can enhance the accuracy of LLM responses to medical queries.