Performance of GPT-4o and Claude in Medical Ethics Scenarios: Comparative Study

Area of Science:

• Artificial Intelligence in Medicine
• Medical Ethics Education
• Large Language Models (LLMs)

Background:

• The application of AI, particularly LLMs, in medicine is rapidly evolving.
• Limited research evaluates AI model performance in medical ethics scenarios, especially specialty-specific ones.

Purpose of the Study:

• To compare the performance of GPT-4o and Claude Sonnet 4 on medical ethics questions.
• To benchmark AI performance against medical students and orthopedic residents.

Main Methods:

• 200 ethical/legal scenario questions from medical student and orthopedic resident question banks were used.
• Questions were administered to GPT-4o and Claude Sonnet 4 three times each.
• Performance was compared to average human test-taker scores.

Main Results:

• Claude Sonnet 4 achieved 89% accuracy, significantly outperforming GPT-4o (70%) and human test-takers (71%).
• GPT-4o exhibited higher response variability than Claude Sonnet 4, particularly on general questions.
• Claude Sonnet 4 demonstrated superior performance across most question categories.

Conclusions:

• Claude AI shows significantly higher proficiency in medical ethics scenarios than GPT-4o and human counterparts.
• AI model architecture and training data likely influence performance differences.
• Further research is crucial to address AI output variability before clinical integration.