Incidental Prompt Injections on Vision-Language Models in Real-Life Histopathology

Vision-language models (VLMs) can analyze multimodal medical data. However, a significant weakness of VLMs is their susceptibility to prompt injection attacks. Here, the model receives conflicting instructions, leading to potentially harmful outputs. We hypothesized that handwritten labels and watermarks on histopathological images could act as inadvertent prompt injections, influencing decision-making in histopathology. We conducted a quantitative analysis with a total of 3888 observations on the state-of-the-art VLMs Claude 3 Opus, Claude 3.5 Sonnet, and GPT-4o. We designed various real-world-inspired scenarios in which we investigated how VLMs react to different labels and watermarks if presented with those next to the tissue. All models reached almost perfect accuracies (90 to 100%) for ground-truth-leaking labels and abysmal accuracies (0 to 10%) for misleading labels or watermarks, despite baseline accuracies between 30 and 65% for various multiclass problems. All VLMs relied on the human-provided labels and accepted them as infallible, even when those inputs contained obvious errors. Furthermore, prompt engineering could not mitigate these effects. We therefore provide evidence that labels, pen- and watermarks on histopathological images act as inadvertent prompt injections leading to blatant mistakes in decision-making by state-of-the-art VLMs. Our findings reveal a critical vulnerability that could compromise diagnostic accuracy and patient safety in a real-world setting. Future research should focus on developing robust methods to detect and neutralize potential prompt injections.