Related Experiment Video
Updated: Jul 9, 2026
Superior Auto-Identification of Trypanosome Parasites by Using a Hybrid Deep-Learning Model
Published on: October 27, 2023
From setting to vetting: using artificial intelligence for Single Best Answer questions review
Olivia Ng1, Siew Ping Han1, Magdalene Hui Min Lee2
1Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore.
Purpose:
Maintaining the quality of Single Best Answer (SBA) questions remains a challenge in medical education, especially as artificial intelligence (AI)-generated items become more common. While considerable attention has been paid to AI question generation, the vetting process is under-explored and difficult to scale.
Methods:
This study investigates the feasibility and reliability of using a large language model to support the vetting of SBA questions. An AI-based reviewer, QA-bot, was developed using custom GPT and embedded with 25 criteria aligned with Bloom's taxonomy (Levels 1-3). QA-bot and two experienced educators independently evaluated 32 AI-generated SBA questions using the shared evaluation rubric.
Results:
The rubric showed high internal consistency (Cronbach's alpha=0.878), and strong inter-rater reliability between human reviewers (intraclass correlation coefficient [ICC]=0.893). QA-bot demonstrated good alignment with human raters (ICC=0.861 and 0.840). While the AI performed well on objective, rule-based criteria, it was less consistent in detecting irrelevant complexity and accurately judging difficulty.
Conclusion:
These findings suggest that AI can function as an efficient first-pass reviewer, improving consistency and reducing workload, with human oversight remaining essential for educational and clinical relevance.
