Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Healthcare Technology

Background:

• Large Language Models (LLMs) demonstrate advanced capabilities, driving interest in their application within healthcare.
• However, a systematic characterization of the risks associated with LLMs in medical contexts is lacking.
• Ensuring safe and trustworthy AI is paramount for clinical adoption.

Purpose of the Study:

• To systematically characterize the risks of LLMs in medical applications.
• To propose a framework for safe and trustworthy medical AI, including five key principles: Truthfulness, Resilience, Fairness, Robustness, and Privacy.
• To introduce the MedGuard benchmark for evaluating LLM performance in healthcare.

Main Methods:

• Development of a comprehensive framework for medical AI safety.
• Creation of the MedGuard benchmark, comprising 1,000 expert-verified medical questions.
• Evaluation of 11 commonly used LLMs against the MedGuard benchmark.

Main Results:

• Current LLMs generally exhibit poor performance across most MedGuard benchmarks, even with safety alignment.
• Performance of LLMs significantly lags behind that of human physicians in medical tasks.
• A notable safety gap exists between current LLMs and the requirements for safe clinical deployment.

Conclusions:

• Despite LLMs' potential, a significant safety gap hinders their widespread adoption in healthcare.
• Human oversight and robust AI safety guardrails are crucial for mitigating risks.
• Further research and development are needed to enhance the safety and reliability of LLMs in medical applications.