Introducing AI as members of script concordance test expert reference panel: A comparative analysis

The Script Concordance Test (SCT) is increasingly used in professional development to assess clinical reasoning, with linear progression in SCT performance observed as clinical experience increases. One challenge in implementing SCT is the potential burnout of expert reference panel (ERP) members. To address this, we introduced ChatGPT as panel members. The aim was to enhance the efficiency of SCT creation while maintaining educational content quality and to explore the effectiveness of different models as reference panels.

A quasi-experimental comparative design was employed, involving all undergraduate medical students and faculty members enrolled in the Ophthalmology clerkship. Two groups involved Traditional ERP which consisted of 15 experts, diversified in clinical experience: 5 senior residents, 5 lecturers, and 5 professors and AI-Generated ERP which is a panel generated using ChatGPT and o1 preview, designed to mirror diverse clinical opinions based on varying experience levels.

Experts consistently achieved the highest mean scores across most vignettes, with ChatGPT-4 and o1 scores generally slightly lower. Notably, the o1 mean scores were closer to those of experts compared to ChatGPT-4. Significant differences were observed between ChatGPT-4 and o1 scores in certain vignettes. These values indicate a strong level of consistency, suggesting that both experts and AI models provided highly reliable ratings.

These findings suggest that while AI models cannot replace human experts, they can be effectively used to train students, enhance reasoning skills, and help narrow the gap between student and expert performance.