Large Language Models as Tools to Generate Radiology Board-Style Multiple-Choice Questions

Area of Science:

• Artificial Intelligence in Medical Education
• Radiology Education Technology

Background:

• Radiology board-style multiple choice questions (MCQs) are crucial for resident assessment and learning.
• Developing high-quality MCQs requires significant educator time and expertise.
• Large language models (LLMs) offer a potential solution for efficient MCQ generation.

Purpose of the Study:

• To evaluate the capability of LLMs in generating radiology board-style MCQs, answers, and rationales.
• To compare the quality of LLM-generated MCQs with existing board examination items.

Main Methods:

• Two LLMs, Llama 2 and GPT-4, were used to create 104 MCQs based on the American Board of Radiology exam blueprint.
• Board-certified radiologists assessed LLM-generated MCQs and prior American College of Radiology (ACR) Diagnostic Radiology In-Training (DXIT) exam items on clarity, relevance, difficulty, distractors, and rationale adequacy.
• Radiologists were blinded to the source of the MCQs during the assessment.

Main Results:

• GPT-4 generated MCQs received scores comparable to ACR DXIT items across all assessed criteria, with near-perfect ratings.
• Llama 2-generated MCQs scored lower than ACR DXIT items but were still considered suitable.
• GPT-4 achieved 100% accuracy in answer generation, while Llama 2 achieved 69% accuracy.

Conclusions:

• Advanced LLMs like GPT-4 can effectively generate high-quality radiology board-style MCQs and rationales.
• LLMs can serve as valuable tools for radiology educators to enhance exam preparation materials and expand question banks.
• The use of LLMs may facilitate the broader application of MCQs as teaching and learning instruments in radiology.