Prompt-Induced Diagnostic Bias in Large Language Model Classification of Echocardiography Reports

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Cardiology

Background:

• Large language models (LLMs) show promise for analyzing medical data.
• Prompt engineering can influence LLM outputs, potentially introducing bias.
• Accurate classification of left ventricular ejection fraction (LVEF) is crucial for cardiovascular diagnosis.

Purpose of the Study:

• To assess if embedding diagnostic expectations in prompts biases LLM classification of LVEF from echocardiography reports.
• To determine if prompt-level mitigation strategies can counteract such biases.

Main Methods:

• GPT-5 was tested using 1,500 structured echocardiography reports.
• Evaluated 1 baseline, 3 bias-injected, and 1 mitigation prompt condition.
• Analyzed shifts in LVEF classification accuracy across different prompt strategies.

Main Results:

• Bias-injected prompts significantly altered LVEF classifications across all categories.
• Classification shifts were directionally consistent with the biased expectations.
• The instruction-based mitigation strategy successfully restored classification accuracy near baseline levels.

Conclusions:

• Prompt-induced bias presents a significant risk to the accuracy of diagnostic classification by LLMs.
• Prompt-level safeguards are effective in mitigating bias and ensuring reliable LLM performance.
• LLM deployment in clinical settings may be supported by implementing prompt-level safeguards.