Evaluating the Performance of GPT-4.0 in Generating and Refining Multiple-Choice Questions for Dental Education

The integration of large-language models into medical education assessment holds transformative potential, yet rigorous evaluation of their capabilities in generating specialized examination content remains underexplored. This study aims to explore the performance of GPT-4.0 in crafting multiple-choice questions (MCQs) within the domain of oral and maxillofacial surgery (OMFS), benchmarking it against GPT-3.5 and GPT-4o, and evaluates its capacity to assess and revise defective items.

100 MCQs generated by GPT-4.0 were evaluated on six aspects with 5-point scales. Comparative analyses were conducted among ChatGPT 4.0, ChatGPT 3.5, and ChatGPT 4o. Additionally, 43 defective MCQs generated by GPTs were input into GPT-4.0 to assess its capabilities in detecting and providing modification suggestions. Inter-rater reliability between the two expert reviewers was assessed using Cohen's kappa coefficient.

GPT-4.0 generated questions rapidly and efficiently, with 87% of them being of high quality. However, 6% required replacement due to issues such as interdisciplinary ambiguity, flawed distractors, or item duplication, while an additional 7% needed minor revisions before use. The comparative analysis highlighted GPT-4.0's marked advantage over GPT-3.5, which exhibited higher defect rates and disciplinary misalignments. Notably, GPT-4.0's ability to detect and revise defective questions proved limited, excelling only in identifying duplicate items but faltering in recognizing disciplinary overlaps or erroneous answers.

GPT-4.0 proves to be a highly capable tool for generating high-quality OMFS MCQs, outperforming both GPT-3.5 and GPT-4o. Nevertheless, its limited capacity to detect nuanced medical flaws and distinguish between closely related disciplines necessitates mandatory human expert oversight to ensure the accuracy and validity of medical assessments.