AI-assisted script concordance tests: Enhancing feasibility with customized ChatGPT

The Script Concordance Test (SCT) assesses clinical reasoning by evaluating responses to uncertain scenarios against expert clinician panels. Unlike traditional multiple-choice questions (MCQs), SCTs measure how examinees structure their knowledge in complex, evolving contexts. Grounded in script theory, SCTs capture the cognitive networks clinicians develop through experience, allowing assessment of real-world decision-making. Developing SCTs presents challenges, including crafting clinically relevant scenarios with appropriate ambiguity and ensuring expert panel reliability. Scoring depends on concordance with expert panels, requiring careful recruitment of 15-20 experts for validity. These logistical demands complicate SCT implementation, particularly in high-stakes assessments.

To address these challenges, we leveraged artificial intelligence (AI), utilizing ChatGPT to generate and score SCTs in ophthalmology. We refined prompts to emulate medical educators, producing SCT vignettes aligned with curricular blueprints. A customized ChatGPT system was trained to assist SCT development, incorporating expert-derived scoring keys. We created one SCT test composed of 10 questions, each with three items assessed through a 5-point Likert scale.

ChatGPT-generated SCTs effectively simulate clinical scenarios, structure scoring, and analyze response patterns. Future work will expand AI-assisted SCTs to other specialties, creating an archive of validated vignettes.