Artificial Intelligence-Assisted Generation of Case Scenarios and Multiple-Choice Questions in Psychiatry: A Pilot...

This study examined the potential of ChatGPT-4, a large language model (LLM), to generate case-based multiple-choice questions (MCQs) in undergraduate psychiatry education and assessed the quality and clinical relevance of the outputs through expert evaluation.

Twelve psychiatric diagnoses commonly encountered in clinical practice were selected. For each diagnosis, a clinical vignette and five MCQs addressing diagnosis, treatment, differential diagnosis, prognosis, and complications were generated using ChatGPT-4, based on structured prompts. The resulting 12 vignettes and 60 questions were divided into six forms. Each form was evaluated by two psychiatrists via an online survey. Evaluators responded to the questions, reviewed the suggested correct answers, and rated the quality and relevance of the items and vignettes using a standardized form. Evaluators also indicated whether each question assessed clinical reasoning or rote knowledge.

Twelve psychiatrists completed evaluations. A majority of the items (71%) were rated as having an appropriate level of difficulty, and 72% of distractors were deemed plausible. However, 28% of questions were reported to contain more than one seemingly correct answer. Clinical vignettes were judged to be representative and diagnostically relevant in 23 out of 24 evaluations. On average, 81.5% of the ChatGPT-selected correct answers were chosen by evaluators. Most reviewers (21/24) stated that the questions assessed clinical reasoning.

ChatGPT-4 can generate clinically relevant vignettes and MCQs that may support case-based learning in psychiatry education. While promising, expert oversight remains essential, especially in generating questions targeting deeper clinical judgment and minimizing ambiguity in answer options.