First, do NOHARM: towards clinically safe large language models.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Clinical Safety

Background:

• Large language models (LLMs) are increasingly used for medical advice by both physicians and patients.
• The clinical safety profiles of LLM-generated medical advice are not well understood.
• Existing benchmarks do not adequately assess the potential harm from AI in medical contexts.

Purpose of the Study:

• To introduce NOHARM (Numerous Options Harm Assessment for Risk in Medicine), a novel benchmark for evaluating the clinical safety of LLM-generated medical recommendations.
• To quantify the frequency and severity of harm associated with LLM advice across various medical specialties.
• To assess the correlation between LLM safety performance and existing AI/medical knowledge benchmarks.

Main Methods:

• Developed NOHARM using 100 real primary care-to-specialist consultation cases spanning 10 specialties.
• Collected 12,747 expert annotations on 4,249 clinical management options generated by 31 LLMs.
• Analyzed the frequency and severity of harm, distinguishing between errors of commission and omission.

Main Results:

• LLM recommendations pose a risk of severe harm in up to 22.2% of cases.
• Harm of omission constitutes the majority of errors, accounting for 76.6% of all identified harms.
• LLM safety performance showed only moderate correlation (r = 0.61-0.64) with existing benchmarks.
• The best-performing LLMs demonstrated superior safety compared to generalist physicians.
• A multi-agent approach using diverse models improved safety over solo LLM performance.

Conclusions:

• Despite proficiency in existing evaluations, widely used LLMs can generate severely harmful medical advice at significant rates.
• Clinical safety must be recognized as a distinct and critical performance dimension for medical AI, requiring explicit measurement.
• The NOHARM benchmark provides a crucial tool for assessing and improving the safety of AI in healthcare.