Encouragement vs. liability: How prompt engineering influences ChatGPT-4's radiology exam performance

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging and Radiology

Background:

• Large Language Models (LLM) show potential in healthcare, particularly radiology.
• Current LLM performance in image-based medical scenarios requires improvement.

Purpose of the Study:

• To evaluate the effect of prompt engineering on ChatGPT-4's accuracy for diagnostic radiology.
• To assess LLM performance on textual and visual questions from the American College of Radiology In Training Test.

Main Methods:

• Developed four distinct personas with unique prompts for ChatGPT-4.
• Evaluated ChatGPT-4's responses to the 2022 American College of Radiology In Training Test questions.
• Compared accuracy and response rates across different personas.

Main Results:

• Encouraging prompts and those disclaiming responsibility yielded higher accuracy.
• Personas involving legal threats or clinical responsibility resulted in lower scores and higher refusal rates.
• Prompt context is crucial for optimizing LLM performance in medical applications.

Conclusions:

• Prompt engineering strategies can significantly influence LLM accuracy in medical contexts.
• Responsible AI integration requires careful consideration of prompt design.
• Further research is needed to optimize LLM use in radiology and medical practice.