Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Medical knowledge is context-dependent, requiring consistent reasoning across semantic variations in natural language.
• Patient communication often uses brand-name drugs (e.g., Advil, Tylenol) instead of generic equivalents, posing challenges for medical AI.
• Evaluating the robustness of medical AI to these variations is critical for reliable clinical applications.

Purpose of the Study:

• To create a novel robustness dataset, RABBITS, for evaluating LLM performance on medical benchmarks.
• To assess the impact of substituting brand and generic drug names on LLM accuracy.
• To identify potential causes for performance degradation in medical LLMs.

Main Methods:

• Developed the RABBITS dataset with physician expert annotations for brand-generic drug name substitutions.
• Evaluated open-source and API-based Large Language Models (LLMs) on established medical question-answering datasets (MedQA, MedMCQA).
• Analyzed performance differences before and after drug name substitutions.

Main Results:

• A consistent performance drop, ranging from 1-10%, was observed in LLMs after drug name swapping.
• Both open-source and API-based LLMs exhibited this fragility.
• Analysis suggested potential test data contamination in pre-training datasets as a contributing factor.

Conclusions:

• LLMs demonstrate fragility in medical question-answering tasks due to variations in drug nomenclature.
• The RABBITS dataset provides a valuable resource for assessing and improving LLM robustness in healthcare.
• Mitigating test data contamination and enhancing contextual understanding are crucial for reliable medical AI.