Patient Cognitive Bias in Large Language Model-Supported Health Consultations: Simulation-Based Comparative Study

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Clinical Decision Support

Background:

• Large language models (LLMs) are increasingly utilized by patients for health information and preliminary medical advice.
• Patient input in LLM consultations can be cognitively biased, emphasizing preferred diagnoses or explanations.
• This bias can constrain the diagnostic context and steer LLM reasoning in health consultations.

Purpose of the Study:

• To quantify the impact of patient cognitive bias on LLM diagnostic performance in multiturn consultations.
• To assess the effectiveness of prompt-based mitigation and decoding temperature adjustments.
• To evaluate a dual-system framework for enhancing LLM robustness against biased interactions.

Main Methods:

• A simulated patient agent generated unbiased and biased consultations using US Medical Licensing Examination cases.
• Six LLMs were evaluated in 3-round dialogues, assessing diagnostic accuracy.
• Prompt strategies, decoding temperatures, and a dual-system framework (conversational LLM + reasoning LLM) were tested.

Main Results:

• Cognitive bias reduced diagnostic accuracy by 7-39 percentage points across models, particularly impacting lower-capacity LLMs.
• Prompt strategies and temperature adjustments showed limited effectiveness in mitigating bias.
• The dual-system framework significantly improved accuracy under bias (10-39 percentage points), recovering performance lost to bias.

Conclusions:

• Patient cognitive bias poses a significant risk in LLM-supported health consultations.
• Standard mitigation techniques offer limited resilience against bias.
• A dual-system framework separating interaction and reasoning enhances diagnostic robustness and offers a scalable design for safer medical AI.