Leveraging large language models to generate multiple-choice questions for epilepsy education

Large language models (LLMs) are increasingly used to generate medical assessment items, but their reliability for disease-specific question writing remains uncertain. We benchmarked seven LLMs for epilepsy single-best-answer multiple-choice question generation under a standardized blueprint, focusing on structural compliance, clinical accuracy, expert-rated quality and educational usability.

Each model completed the same 60 prespecified tasks spanning foundational knowledge, clinical cases, treatment decision-making, and diagnostic evaluation and longitudinal management, yielding 420 questions. Tasks were delivered through an identical single-turn prompt without model-specific optimization, web search or external tools. Objective evaluation assessed structural compliance, keyed-answer accuracy, textual characteristics and response time. Two blinded clinical reviewers independently rated content rigor, clarity, distractor quality, educational usefulness, cognitive alignment and overall usability on five-point scales.

After one standardized corrective prompt, all 420 questions met structural requirements, although first-pass compliance ranged from 83.33% to 100%. Keyed-answer accuracy varied substantially, from 61.67% to 98.33%, with Claude-5 achieving the highest accuracy. GPT-5.6 obtained the highest composite expert score (median, 3.75), followed by Claude-5 (3.67) and Kimi-K3 (3.50). Distractor quality was a common limitation, with median scores of only 2-3 across models. Only 2 of 420 questions (0.48%) were judged directly usable without revision, both generated by GPT-5.6; 119 (28.33%) were usable after minor revision and 299 (71.19%) required substantial revision. Longer outputs were associated with better cognitive alignment but not with content rigor or clarity, while response time was unrelated to overall usability.

Contemporary LLMs can reliably reproduce the structure of epilepsy MCQs, but structural compliance and answer accuracy do not ensure educational readiness. Model strengths were multidimensional, and weak distractor construction remained a major barrier to direct use. LLMs may support expert-supervised item drafting, but independent clinical review and subsequent learner-based psychometric validation remain necessary before deployment in medical education.