Disagreements in Medical Ethics Question Answering Between Large Language Models and Physicians

Area of Science:

• Medical Ethics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• Medical ethics involves diverse perspectives, making consistent ethical decision-making challenging.
• Integrating artificial intelligence, specifically large language models (LLMs), into healthcare necessitates evaluating their ethical reasoning capabilities.

Purpose of the Study:

• To compare the agreement rates of three advanced LLMs (GPT-4, Gemini-pro-1.5, Llama-3-70b) against a multidisciplinary physician group when answering ethically ambiguous medical questions.

Main Methods:

• A cross-sectional study where LLMs generated 1,248 medical ethics questions based on the American College of Physicians Ethics Manual.
• Physicians and LLMs answered generated questions; agreement rates were compared among physicians, between physicians and LLMs, and among LLMs.

Main Results:

• Physicians' agreement rate was 55.9% on moderately complex ethical questions.
• Agreement between physicians and LLMs was 57.9%, while agreement among LLMs was significantly higher at 76.8% (p < 0.001).

Conclusions:

• LLMs exhibit greater consistency than physicians in responding to complex medical ethics scenarios.
• LLMs show potential as consultants for ambiguous ethical situations, enhancing consistency in healthcare decision-making.