Corpus-Based Evaluation of Decision-Making in Medical Ethics by Large Language Models.

Area of Science:

• Medical Ethics
• Artificial Intelligence
• Clinical Informatics

Background:

• Assessing ethical biases in large language models (LLMs) is critical for safe clinical integration.
• Existing evaluations lack comprehensive frameworks for clinical ethical dilemmas.

Purpose of the Study:

• To introduce a novel text corpus for evaluating LLM ethical decision-making in clinical scenarios.
• To assess the ethical performance of advanced LLMs like GPT-4 and GEMINI.

Main Methods:

• Development of a 50-scenario clinical case corpus with ethically dilemmatic options.
• Scoring LLM responses based on four core ethical principles.
• Analysis of LLM decision trajectories and adaptability to instructions.

Main Results:

• GPT-4 and GEMINI significantly outperformed random selection across all ethical principles.
• LLMs demonstrated adaptability but also ethical vulnerabilities when following undesirable instructions.
• Ethical scores varied based on specific case instructions.

Conclusions:

• LLMs show potential for ethical clinical decision-making but require careful oversight.
• The developed corpus provides a robust tool for assessing LLM ethical performance.
• Future work should expand the corpus to include diverse cultural and institutional contexts.