In the search for the perfect prompt in medical AI queries

Area of Science:

• Medical Artificial Intelligence
• Health Informatics
• Clinical Decision Support

Background:

• Evaluating medical Artificial Intelligence (AI) systems is challenging due to performance variability across studies.
• Prompt quality, or how questions are posed to AI, is an under-recognized factor influencing AI outcomes.
• Existing literature shows inconsistencies in AI performance, ranging from surpassing human experts to underperforming in broader analyses.

Purpose of the Study:

• To investigate the impact of prompt engineering on the perceived accuracy and reliability of conversational AI in medicine.
• To identify the critical role of prompt quality in the performance paradox of medical AI.
• To explore the gap between expert-formulated prompts and public queries, addressing safety and health equity.

Main Methods:

• Narrative review of scientific literature.
• Analysis focused on publications from January 2018 to August 2025.
• Investigation into the relationship between prompt characteristics and AI performance metrics.

Main Results:

• A "performance paradox" exists where AI excels in controlled settings but falters in real-world applications, strongly linked to prompt type.
• Prompting bias can invalidate study conclusions, and AI hallucinations pose risks of generating incorrect medical information.
• A significant disparity was found between expert-level prompts and natural public queries, impacting AI's practical utility and safety.

Conclusions:

• Prompt engineering is a critical determinant of medical AI reliability and accuracy.
• Addressing prompting bias and AI hallucinations is essential for trustworthy medical AI.
• Bridging the gap between expert and public prompts is vital for safe, equitable, and effective AI integration in healthcare.