AI for Assessment in Medical Education in Post LLM Era: A Scoping Review

Artificial intelligence (AI) has supported assessment in medical education for decades through automatic item generation and natural language processing (NLP), but these pre-large language model (LLM) approaches were narrow, tool-intensive, and required substantial expert oversight. The release of ChatGPT in November 2022 marked a paradigm shift, enabling rapid generation and evaluation of assessment content. This scoping review aimed to map post-November-2022 evidence on the use of AI, particularly LLMs, in designing and evaluating assessments in medical education. This review was conducted in accordance with PRISMA-ScR guidelines. PubMed and Scopus were systematically searched for literature published from November 2022 to July 2025 using keywords related to artificial intelligence, large language models, assessment, and medical education. Original empirical studies examining AI-supported assessment design or evaluation for medical learners were included. Twenty-five studies met the inclusion criteria. Two dominant application domains emerged. First, AI-assisted generation and evaluation of multiple-choice questions (MCQs) across medical disciplines demonstrated efficiency gains, with acceptable difficulty indices in some examinations. However, consistent limitations were identified, including shallow reasoning, weak distractors, factual inaccuracies, and reduced discriminatory power, necessitating expert review. Second, the use of NLP and LLMs for analysis of narrative feedback showed increasing maturity, enabling theme extraction, quality scoring, risk prediction, and dashboard-based visualization to support formative assessment. Overall, the literature supports AI as an assistive rather than substitutive tool in assessment. Future research should move from feasibility to assurance by standardizing human-in-the-loop workflows, implementing mandatory post-exam psychometric analyses, developing multimodal assessment pipelines, and conducting multi-institutional trials to evaluate educational impact.