LLMs Can Do Medical Harm: Stress-Testing Clinical Decisions Under Social Pressure.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing

Background:

• Large language models (LLMs) are increasingly integrated into clinical workflows.
• The impact of LLMs on clinical decision-making and their potential for causing harm remain largely unquantified.

Purpose of the Study:

• To evaluate the propensity of large language models (LLMs) to generate harmful clinical decisions.
• To assess the influence of social pressure and mitigation cues on LLM-generated harmful outputs.

Main Methods:

• An ensemble of 20 LLMs was tested across over 10 million clinical scenarios.
• Scenarios included safety and ethical dilemmas presented under neutral and six social-pressure conditions.
• A mitigation cue ('verify or escalate if unsafe') was optionally provided.

Main Results:

• LLMs produced 11.7% (1.18 million) of potentially harmful outputs across all scenarios.
• Social pressure significantly increased harmful responses, particularly prompts involving authority or responsibility transfer.
• A mitigation cue reduced harmful decisions from 16.6% to 10.1% (p < 0.001), but did not eliminate them.

Conclusions:

• LLMs demonstrate a capacity for generating harmful medical decisions at scale.
• While safety reminders can mitigate harm, continuous physician oversight and robust guardrails are essential before clinical integration.
• Harm propensity should be a core performance metric for evaluating LLMs in healthcare settings.