Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• High-stakes domains like medicine require models to adhere strictly to provided context.
• However, challenges arise when context conflicts with established model priors or safety guidelines.

Purpose of the Study:

• To investigate the behavior and reasoning of large language models (LLMs) when presented with counterfactual or adversarial medical evidence.
• To assess the reliability and safety of LLMs in medical question-answering tasks under manipulated evidence conditions.

Main Methods:

• Construction of MedCounterFact, a novel counterfactual medical question-answering dataset.
• Systematic replacement of real-world medical interventions with four types of counterfactual stimuli (e.g., unknown words, toxic substances).
• Evaluation of multiple frontier LLMs using clinical comparison questions based on randomized controlled trials.

Main Results:

• Frontier LLMs overwhelmingly accepted counterfactual medical evidence at face value, even when dangerous or implausible.
• Models provided confident and uncaveated answers despite contradictory or harmful information.
• A significant overemphasis on faithfulness to context over safety was observed.

Conclusions:

• Current LLMs struggle to identify and reject unsafe or implausible medical information presented as context.
• There is a critical need to develop LLM safety protocols that prioritize safety over mere faithfulness in medical applications.
• Future research should focus on enabling LLMs to critically evaluate evidence and align with safety standards.