Teaching Clinical Reasoning in Health Care Professions Learners Using AI-Generated Script Concordance Tests: Mixed...

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Clinical Reasoning Assessment

Background:

• Artificial intelligence (AI) is transforming medical education tools for teaching and assessment.
• Script concordance tests (SCTs) are effective for evaluating clinical reasoning under uncertainty.
• Traditional SCTs rely on resource-intensive expert panels for scoring and feedback.

Purpose of the Study:

• To evaluate the efficacy of large language models (LLMs) in simulating expert judgment for SCTs.
• To assess AI's capability in authoring, scoring, and providing feedback for cardiology and pneumology SCTs.
• To gauge student perceptions of SCT difficulty and the pedagogical value of AI-generated feedback.

Main Methods:

• A cross-sectional, mixed-methods study involving 25 medical students.
• A 32-item SCT authored by ChatGPT-4o was administered.
• Six LLMs (trained and untrained) served as simulated experts for scoring and feedback generation.

Main Results:

• Trained AI models demonstrated significantly higher concordance (ρ=0.64) with student responses compared to untrained models (ρ=0.41).
• AI-generated feedback was rated as most helpful in 62.5% of cases, particularly from trained models.
• The SCT showed good internal consistency (Cronbach α=0.76), with moderate perceived difficulty by students.

Conclusions:

• Trained generative AI models can reliably simulate expert clinical reasoning in SCTs.
• AI offers a potential solution to streamline SCT design and provide authentic, educationally valuable feedback.
• Future research should explore AI's longitudinal impact on learning and hybrid human-AI models for medical education.