Assessing LLM-generated vs. expert-created clinical anatomy MCQs: a student perception-based comparative study in...

Area of Science:

• Medical Education
• Artificial Intelligence in Education
• Anatomy Assessment

Background:

• Large language models (LLMs) are increasingly utilized for generating educational content in medical settings.
• The efficacy of LLM-generated multiple-choice questions (MCQs) compared to expert-authored questions in anatomy education is not well-established.

Purpose of the Study:

• To compare the quality and effectiveness of MCQs generated by ChatGPT, Google-Gemini, and a clinical anatomist.
• To evaluate learner perceptions of AI-generated versus expert-written anatomy MCQs.

Main Methods:

• A cross-sectional, mixed-methods study involving 48 medical students.
• Participants completed and evaluated three sets of anonymized MCQs (ChatGPT, Gemini, expert-authored) based on 17 quality criteria.
• Descriptive statistics, chi-square analysis, and thematic review of qualitative feedback were employed.

Main Results:

• Most students perceived MCQs from all three sources as equally effective.
• ChatGPT was frequently preferred for identifying knowledge gaps via challenging distractors.
• Expert-written questions were rated highest for promoting deeper analytical thinking; significant preference variations were noted (χ²(64) = 688.79, p < .001).

Conclusions:

• LLM-generated anatomy MCQs demonstrate comparable learner-perceived value to expert-authored questions.
• AI-generated MCQs can enhance engagement, but expert oversight is crucial for clarity and curricular alignment.
• A hybrid AI-human approach offers a scalable solution for high-quality medical education assessment design.