Medical large language models are vulnerable to data-poisoning attacks.

Area of Science:

• Artificial Intelligence
• Biomedical Informatics
• Data Science

Background:

• Large language models (LLMs) are increasingly adopted in healthcare.
• LLMs trained on internet data risk ingesting and propagating medical misinformation.
• Existing benchmarks may not detect subtle harms from poisoned training data.

Purpose of the Study:

• To assess the risk of medical misinformation propagation by LLMs trained on poisoned data.
• To evaluate the effectiveness of standard benchmarks in detecting LLM harm from data poisoning.
• To propose and validate a harm mitigation strategy using biomedical knowledge graphs.

Main Methods:

• Simulated a data-poisoning attack on "The Pile" dataset, a common LLM training resource.
• Evaluated corrupted LLM performance against standard open-source benchmarks.
• Developed and tested a harm mitigation algorithm using biomedical knowledge graphs to screen LLM outputs.

Main Results:

• Replacing just 0.001% of training tokens with misinformation created harmful LLMs.
• Corrupted LLMs performed comparably to uncorrupted ones on standard benchmarks.
• The proposed knowledge graph-based strategy captured 91.9% of harmful content (F1 = 85.7%).

Conclusions:

• LLMs trained on web-scraped data pose significant risks for medical misinformation in healthcare.
• Standard LLM evaluation benchmarks are insufficient for detecting data-poisoning-induced medical harms.
• Biomedical knowledge graphs offer a promising method for validating LLM outputs and mitigating safety risks.