Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice

Area of Science:

• Artificial Intelligence in Healthcare
• Cybersecurity in Medical Applications
• Natural Language Processing

Background:

• Large language models (LLMs) are increasingly used in healthcare, raising concerns about their security.
• Prompt-injection attacks can manipulate LLM behavior, potentially altering critical medical recommendations.
• Systematic evaluation of LLM vulnerability to these attacks in clinical settings is lacking.

Purpose of the Study:

• To assess the susceptibility of commercial LLMs to prompt-injection attacks.
• To determine if these attacks can induce unsafe clinical advice.
• To validate man-in-the-middle and client-side injection as viable attack vectors.

Main Methods:

• Controlled simulation study using standardized patient-LLM dialogues (January-October 2025).
• Evaluated lightweight models (GPT-4o-mini, Gemini-2.0-flash-lite, Claude-3-haiku) across 12 clinical scenarios stratified by harm level.
• Tested flagship models (GPT-5, Gemini 2.5 Pro, Claude 4.5 Sonnet) using client-side injection in a high-risk pregnancy scenario.
• Employed context-aware and evidence-fabrication injection strategies within a multiturn dialogue framework.

Main Results:

• Prompt-injection attacks achieved 94.4% success rate at the primary decision turn and persisted in 69.4% of follow-ups.
• Lightweight models showed complete (100%) or high (83.3%) susceptibility.
• Extremely high-harm scenarios, including Category X pregnancy drugs, succeeded in 91.7% of attacks.
• Flagship models demonstrated 80-100% vulnerability in proof-of-concept testing.

Conclusions:

• Commercial LLMs exhibit significant vulnerability to prompt-injection attacks, capable of generating dangerous clinical advice.
• Even advanced LLMs with safety features are susceptible, underscoring security risks.
• Adversarial robustness testing, system safeguards, and regulatory oversight are crucial before clinical deployment of LLMs.