Generating Image-Based Multiple-Choice Questions with Multimodal Large Language Models: Expert and Psychometric...

The purpose of this study is to evaluate the expert-rated quality and item-level psychometric performance of multiple-choice questions generated by GPT-4o and o3 from correctly recognized radiographs and compare selected items with faculty-written questions. Fifty curriculum-aligned radiographs were submitted separately to both models without clinical information. Only questions for which two radiologists agreed that the generated content correctly reflected the reference diagnosis proceeded to expert review. Seventy-two artificial intelligence-generated questions were evaluated by 12 radiology experts. A 36-item assessment containing 12 GPT-4o, 12 o3, and 12 faculty-written questions, balanced across intended Bloom levels, was administered to 112 fifth-year medical students. Item difficulty, corrected item-total correlations, and distractor functionality were evaluated. Each model correctly recognized 12 of 50 radiographs (24%). Expert ratings were generally favourable. No statistically significant overall difficulty difference was detected across sources (P = 0.379), although equivalence was not assessed. Overall item discrimination differed across sources (P = 0.014); in the post hoc comparison, o3 items showed higher observed discrimination than faculty-written items after Bonferroni adjustment (adjusted P = 0.009). Non-functioning distractors occurred in 16/48 faculty-written, 14/48 GPT-4o, and 11/48 o3 distractors; the exploratory comparison was not significant (P = 0.629). Findings reflect downstream question generation conditional on successful image recognition, not autonomous generation from unselected radiographs. The observed source differences are preliminary because item sets and selection procedures were not equivalent. Multimodal large language models are best positioned as supervised tools requiring radiologist verification and item-level psychometric evaluation.