Comparative analysis of large language model and faculty rater scoring of medical student documentation in objective...

To evaluate whether a commercially available large language model (LLM) can score medical student post-encounter notes from Objective Structured Clinical Examinations (OSCEs) using an existing institutional rubric without task-specific model training and to characterize internal consistency and alignment with faculty scores using absolute agreement and rank-order association metrics.

The dataset included 433 medical student notes from 7 standardized OSCE cases at the University of Kentucky. In November 2024, an LLM (OpenAI GPT-4o mini, via API) scored each note using a prompt-based workflow (no model fine-tuning) under two approaches: numerical score only ("GPT-NE", 30 iterations per note) and numerical score with rationale ("GPT-E", 4 iterations per note). The authors compared the LLM's performance against historical faculty grades using the same rubric, evaluating the reliability of grading methodologies and inter-modal agreement.

The LLM demonstrated high internal consistency for GPT-NE (intraclass correlation (ICC) = 0.94) and moderate-to-high consistency for GPT-E (ICC = 0.76). LLM scores were significantly lower than faculty scores (mean differences on a 1-4 scale: -1.12 GPT-NE, 95% CI -1.17 to -1.08; -0.65 GPT-E, 95% CI -0.69 to -0.61, both P < .001). Spearman rank-order correlations (r = 0.43-0.53) indicate moderate preservation of relative student ranking despite differences in absolute scores.

The LLM demonstrated significant potential for automating OSCE note grading given its internal consistency and similar ranking of learners compared with faculty raters. Automation of grading could yield substantial time savings; however, discrepancies with absolute faculty scores underscore the need for further refinement of rubric alignment and grading methodologies.