Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions

Area of Science:

• Medical Education
• Artificial Intelligence in Education

Background:

• Creating high-quality medical exam questions is labor-intensive.
• Test-enhanced learning improves student outcomes.
• Automated question generation using large language models (LLMs) could be beneficial but lacks comparative studies.

Purpose of the Study:

• To compare student performance on multiple-choice questions (MCQs) generated by ChatGPT (LLM) versus those created by medical educators.
• To evaluate the item difficulty and discriminatory power of LLM-generated versus human-generated MCQs.

Main Methods:

• Two sets of 25 MCQs each were created: one by a medical educator, the other by ChatGPT 3.5.
• 161 students completed a formative test with 46 MCQs (25 human, 21 LLM) before their neurophysiology exam.
• Students indicated their perceived source (human or LLM) for each question.

Main Results:

• No significant difference in item difficulty was observed between human and LLM questions.
• Human-generated questions demonstrated statistically significantly higher discriminatory power (mean=0.36) than LLM-generated questions (mean=0.24).
• Students correctly identified the source of questions only 57% of the time.

Conclusions:

• While LLMs can generate medical exam questions, human-created questions currently exhibit superior discriminatory power.
• Further research is needed to explore LLM capabilities with different question types and in diverse educational contexts.