A Pilot Study Using Artificial Intelligence to Enhance Efficiency, Accuracy, and Objectivity in Grading Pharmacy...

The goal of this project was to evaluate the feasibility of using artificial intelligence (AI) in grading pharmacy Objective Structured Clinical Examination (OSCE) analytical checklists in terms of accuracy, objectivity and consistency, and efficiency compared to faculty evaluators.

Third-year pharmacy students (n = 39) enrolled in a private Christian university completed a 5-station OSCE as part of the Advanced Pharmacy Practice Experience-readiness plan. Audio recordings from 2 of the interactive stations were de-identified and fed into 2 customized language learning models: a speech-to-text and a tailored transformer model trained on the analytical checklist. A validation set using the analytical checklist was completed by the study investigator. Comparison of AI scoring of the analytical checklist against the validation set and faculty evaluators' scoring was retrospectively reviewed for analysis.

The customized AI model demonstrated >96% and 94% accuracy for stations A and B, respectively. There was an observed statistically significant inter-rater variability among the faculty evaluators, with one evaluator scoring on average 4 points higher in one station and another evaluator scoring on average one point higher in the second station. For efficiency, the AI model graded 39 students in <5 min, saving time for faculty grading, along with timely feedback to assist in improving future student performance.

Customized AI model outperformed faculty scoring on the pharmacy OSCE analytical checklists of 2 stations in accuracy, objectivity and consistency, and efficiency.