Leveraging Large Language Models to Generate Multiple-Choice Questions for Ophthalmology Education.

Area of Science:

• Ophthalmology Education Technology
• Artificial Intelligence in Medical Assessment
• Medical Examination Development

Background:

• High-quality multiple-choice questions (MCQs) are crucial for ophthalmology residency training and board certification.
• Developing such MCQs is time-consuming and resource-intensive for expert committees.

Purpose of the Study:

• To evaluate the capability of general-domain large language models (LLMs), specifically GPT-4, in generating high-quality, novel, and readable ophthalmology MCQs.
• To compare LLM-generated MCQs against those created by a committee of experienced ophthalmology examination writers.

Main Methods:

• A survey study compared MCQs generated by GPT-4o with those from a human expert committee.
• Ten masked expert ophthalmologists evaluated MCQs based on appropriateness, clarity, relevance, discriminative power, and suitability for trainees using a 10-point Likert scale.
• LLM performance was assessed using string similarity, Flesch Reading Ease for readability, and inter-rater reliability (ICC).

Main Results:

• GPT-4o generated MCQs received median scores comparable to expert-written questions across most criteria (e.g., appropriateness, clarity, relevance).
• LLM-generated MCQs demonstrated low similarity scores (<60) to existing content, indicating novelty.
• Readability scores were similar between LLM-generated and human-expert MCQs, with moderate inter-rater reliability.

Conclusions:

• Large language models show potential for developing ophthalmology board-style MCQs, aiding in the expansion of examination banks.
• The quality, novelty, and readability of LLM-generated MCQs are promising but warrant further assessment.
• LLMs can support ophthalmology residency training by providing a scalable method for MCQ generation.