Benchmarking Large Language Models Against Psychiatry Residents Using Traditional Institutional Assessments

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Psychiatric Training and Assessment

Background:

• The application of AI in healthcare is rapidly advancing, yet its comparative efficacy against medical trainees in psychiatric education is not well-understood.
• This study addresses the gap by evaluating large language models (LLMs) against first-year psychiatry residents in standardized assessments.

Purpose of the Study:

• To compare the performance of advanced AI models with first-year psychiatry residents in theoretical and practical assessments.
• To identify AI's strengths and limitations in the context of psychiatric medical education.

Main Methods:

• Utilized existing examination materials (Theory Papers I & II, OSCEs) from 25 first-year psychiatry residents.
• Administered the same assessments to three AI models: ChatGPT-3.5, Gemini Advanced, and Claude Sonnet.
• Evaluated AI responses using blinded faculty assessment and compared scores statistically with resident performance.

Main Results:

• AI models significantly outperformed residents in theoretical knowledge across both Paper I and Paper II.
• Performance in Objective Structured Clinical Examinations (OSCEs) was comparable for Paper I but varied for Paper II, with one AI model underperforming.
• Excellent inter-rater reliability (ICC: 0.810-0.934) was maintained throughout the evaluation.

Conclusions:

• AI demonstrates superior theoretical knowledge acquisition compared to psychiatry residents.
• Variable OSCE performance indicates current AI limitations in clinical reasoning and contextual understanding.
• Psychiatric education must adapt to integrate AI for knowledge synthesis while focusing on essential human clinical competencies.