Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Medical Ethics

Background:

• The United States Medical Licensing Examination (USMLE) assesses physician competency.
• AI models have been studied for USMLE performance, but soft skills assessment is lacking.
• Evaluating AI in communication, ethics, empathy, and professionalism is crucial for medical practice.

Purpose of the Study:

• To assess the performance of ChatGPT and GPT-4 on USMLE-style soft skills questions.
• To compare AI models' capabilities against past human performance on the AMBOSS platform.
• To evaluate AI consistency and confidence in responding to complex medical scenarios.

Main Methods:

• Utilized 80 USMLE-style soft skills questions from official sources and AMBOSS.
• Administered questions to ChatGPT and GPT-4, employing follow-up queries for consistency checks.
• Compared AI model performance against historical data from AMBOSS users.

Main Results:

• GPT-4 achieved 90% accuracy, significantly outperforming ChatGPT's 62.5%.
• GPT-4 demonstrated higher confidence, with no response revisions, unlike ChatGPT (82.5% revisions).
• GPT-4's performance surpassed that of previous AMBOSS users, showing empathy and professionalism.

Conclusions:

• GPT-4 exhibits strong potential for handling USMLE soft skills questions, exceeding both earlier AI and human benchmarks.
• AI models, particularly GPT-4, demonstrate capacity for empathy and ethical reasoning in medical contexts.
• AI may offer valuable support in addressing the interpersonal and ethical demands of medical practice.