Evaluating the Effectiveness of ChatGPT Versus Human Proctors in Grading Medical Students' Post-OSCE Notes

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare

Background:

• Artificial intelligence (AI) tools show promise in various fields, including medical education.
• The utility of AI for assessing medical students' clinical reasoning in written notes remains underexplored.

Purpose of the Study:

• This study aimed to compare the grading of medical students' clinical notes by ChatGPT-4 against evaluations by a human proctor.

Main Methods:

• 127 subjective, objective, assessment, and plan (SOAP) notes from an objective structured clinical examination were analyzed.
• ChatGPT-4 and a human proctor used the same grading rubric to evaluate notes across history, physical exam, differential diagnosis, and treatment plan.
• Statistical analyses included t tests and chi-squared analysis to compare scores.

Main Results:

• ChatGPT-4 assigned significantly different grades compared to proctors in history, differential diagnosis, and treatment plan (P<.001).
• The largest discrepancy was observed in the treatment plan (Cohen's d=1.25).
• ChatGPT-4 resulted in a higher mean cumulative grade and a greater proportion of honors grades (92.9%) compared to proctor grading (63.8%).

Conclusions:

• ChatGPT-4's grading of student SOAP notes showed statistically significant differences from human proctor evaluations, with notable variations in treatment plan assessment.
• Despite numerical differences, the practical impact on overall grades was small, but AI assigned significantly more honors.
• Medical educators must carefully evaluate AI performance within their specific grading frameworks before implementing AI for summative assessments.