Analyzing Question Characteristics Influencing ChatGPT's Performance in 3000 USMLE®-Style Questions

The potential of artificial intelligence (AI) and large language models like ChatGPT in medical applications is promising, yet its performance requires comprehensive evaluation. This study assessed ChatGPT's capabilities in answering USMLE® Step 2CK questions, analyzing its performance across medical specialties, question types, and difficulty levels in a large-scale question test set to assist question writers in developing AI-resistant exam questions and provide medical students with a realistic understanding of how AI can enhance their active learning.

A total of n=3302 USMLE® Step 2CK practice questions were extracted from the AMBOSS© study platform, excluding 302 image-based questions, leaving 3000 text-based questions for analysis. Questions were manually entered into ChatGPT and its accuracy and performance across various categories and difficulties were evaluated.

ChatGPT answered 57.7% of all questions correctly. Highest performance scores were found in the category "Male Reproductive System" (71.7%) while the lowest were found in the category "Immune System" (46.3%). Lower performance was noted in table-based questions, and a negative correlation was found between question difficulty and performance (r _s=-0.285, p <0.001). Longer questions tended to be answered incorrectly more often (r _s=-0.076, p <0.001), with a significant difference in length of correctly versus incorrectly answered questions.

ChatGPT demonstrated proficiency close to the passing threshold for USMLE® Step 2CK. Performance varied by category, question type, and difficulty. These findings aid medical educators make their exams more AI-proof and inform the integration of AI tools like ChatGPT into teaching strategies. For students, understanding the model's limitations and capabilities ensures it is used as an auxiliary resource to foster active learning rather than abusing it as a study replacement. This study highlights the need for further refinement and improvement in AI models for medical education and decision-making.