AI-assisted script concordance tests: Enhancing feasibility with customized ChatGPT

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Cognitive Science

Background:

• The Script Concordance Test (SCT) evaluates clinical reasoning by comparing examinee responses in uncertain scenarios to expert panels.
• Traditional methods face challenges in creating relevant scenarios and ensuring expert panel reliability, complicating high-stakes assessments.

Purpose of the Study:

• To leverage Artificial Intelligence (AI), specifically ChatGPT, to address the challenges in developing and scoring Script Concordance Tests (SCTs).
• To explore the feasibility of AI-generated SCTs for assessing clinical reasoning in ophthalmology.

Main Methods:

• Utilized ChatGPT with refined prompts to generate SCT vignettes aligned with curricular blueprints, emulating medical educators.
• Developed a customized ChatGPT system trained to assist in SCT creation and scoring, incorporating expert-derived scoring keys.
• Constructed a 10-question SCT in ophthalmology, with each question featuring three items assessed on a 5-point Likert scale.

Main Results:

• ChatGPT-generated SCTs demonstrated effectiveness in simulating clinical scenarios and structuring the scoring process.
• AI assistance facilitated the analysis of examinee response patterns.

Conclusions:

• AI, particularly ChatGPT, offers a viable solution for generating and scoring Script Concordance Tests (SCTs), enhancing clinical reasoning assessment.
• Future efforts will focus on expanding AI-assisted SCTs to other medical specialties and building a repository of validated assessment tools.