Cognitively Biased Prompt Effects on Large Language Model Accuracy for Radiology Board-style Examination Questions.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Cognitive Bias in AI

Background:

• Large language models (LLMs) show promise for radiology tasks.
• The impact of cognitive biases on LLM performance in radiology is not well understood.
• Radiology board-style questions are a critical benchmark for AI evaluation.

Purpose of the Study:

• To assess the vulnerability of LLMs to cognitive biases in radiology.
• To quantify the accuracy degradation caused by authority, complexity, and anchoring bias prompts.
• To evaluate the effectiveness of bias mitigation strategies for LLMs in radiology.

Main Methods:

• Ten LLMs were tested on 400 radiology board-style questions (200 text-based, 200 multimodal).
• Models were prompted under baseline conditions and with authority bias prompts (ABPs), complexity bias prompts (CBPs), and anchoring bias prompts (AnBPs).
• Two mitigation techniques, a prompt bias audit and a one-shot mitigation strategy, were applied.

Main Results:

• LLMs achieved 84.8% accuracy on text and 59.5% on multimodal questions at baseline.
• Biased prompts significantly reduced accuracy: ABP (21.1% text, 44.9% multimodal), CBP (10.1% text, 44.4% multimodal), AnBP (4.4% text, 39.6% multimodal).
• Mitigation strategies improved accuracy, with prompt bias audit (5.6% text, 15.8% multimodal) and one-shot mitigation (4.0% text, 24.9% multimodal).

Conclusions:

• LLMs are susceptible to cognitive biases, impacting their accuracy in radiology applications.
• Targeted prompts exploiting cognitive biases can significantly degrade LLM performance.
• Bias mitigation techniques can enhance LLM robustness and reliability in medical contexts.