Can Generative Artificial Intelligence Reliably Score Open-Ended Question Assessments in Undergraduate Medical...

Area of Science:

• Medical Education
• Artificial Intelligence in Assessment
• Natural Language Processing

Background:

• Open-ended questions (OEQs) are valuable for assessing medical knowledge but pose grading challenges due to time constraints.
• Generative artificial intelligence (AI) presents a potential solution for automating the scoring of OEQs.

Purpose of the Study:

• To evaluate the accuracy and reliability of generative AI (GPT-4) in scoring medical student OEQ exams.
• To determine if AI scoring can achieve comparable inter-rater reliability (IRR) to faculty grading.

Main Methods:

• Medical student OEQ responses were scored by GPT-4 using the Med2Lab platform, integrating case vignettes, questions, and rubrics.
• GPT-4 scores were compared to faculty scores using Cohen's weighted kappa (k_w) for IRR.
• Iterative rubric engineering was performed based on error pattern analysis to enhance AI scoring accuracy over three iterations.

Main Results:

• Substantial IRR (k_w=0.88-0.94) was achieved between faculty and GPT-4 using both analytic and holistic rubrics after three iterations.
• Moderate IRR (k_w=0.54) was observed for one question using a holistic rubric.
• Score discrepancies between AI and faculty were minimal, typically only 1 point.

Conclusions:

• Generative AI, specifically GPT-4, can reliably score medical student OEQ exams.
• Iterative rubric engineering is crucial for maximizing the reliability of AI-based OEQ assessment.
• AI-powered scoring offers a promising approach to enhance efficiency in medical knowledge assessment.