Clinical agents fail silently on patient identity.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Health Data Integrity

Background:

• Large language models (LLMs) are increasingly integrated into clinical workflows, powering agents that interact with electronic health records (EHRs).
• Assessing the safety of these LLM-powered agents requires understanding their ability to detect and respond to patient identity discrepancies.

Purpose of the Study:

• To evaluate the performance of various LLM agents in detecting patient identity faults within a simulated clinical record environment.
• To determine if LLM agents correctly identify and flag tampered patient data, preventing misbinding of information.

Main Methods:

• A simulated EHR environment was created using MIMIC-IV data, with specific tampering methods applied to patient records (header swap, MRN change, age shift).
• Six LLM agents (closed and open-weight) performed 1.2 million tool calls to copy ICD-10-CM codes, with tamper detection defined as withholding a write.
• Performance was measured by the agents' ability to output "UNKNOWN" or omit a Store call for tampered records.

Main Results:

• Most LLM agents failed to detect patient identity inconsistencies, frequently copying codes into tampered records.
• GPT-4.1 demonstrated limited success in detecting header swaps (17.4%) but near-zero detection for subtle changes (MRN, age).
• Other models, including GPT-5-chat, showed minimal to no detection of identity faults, highlighting a significant risk of misbinding patient data.

Conclusions:

• Clinical LLM agents exhibit a critical vulnerability in detecting patient identity inconsistencies, posing a risk of misbinding patient information.
• Safe deployment necessitates explicit identity verification mechanisms and benchmarks that prioritize record integrity alongside data accuracy.
• Future development should focus on enhancing LLM agents' ability to recognize and flag data integrity issues to ensure patient safety.