Fine-Tuned Large Language Models for Generating Multiple-Choice Questions in Anesthesiology: Psychometric Comparison...

Area of Science:

• Medical Education
• Artificial Intelligence in Assessment
• Psychometrics

Background:

• Multiple-choice questions (MCQs) are crucial for standardized medical assessment.
• Developing high-quality MCQs requires subject expertise and rigorous methodology.
• Large language models (LLMs) present opportunities for automated MCQ generation, but evaluations are limited.

Purpose of the Study:

• To assess if a fine-tuned LLM can generate anesthesiology MCQs with psychometric properties comparable to faculty-written items.

Main Methods:

• A fine-tuned GPT-4 model was trained on anesthesiology materials.
• The model generated 15 MCQs, which were analyzed alongside 15 faculty-written MCQs.
• Item analysis followed psychometric standards, comparing difficulty, point-biserial correlation, and discrimination index.

Main Results:

• No significant differences were found in difficulty, point-biserial correlation, or discrimination index between LLM-generated and faculty-written MCQs.
• Both sets of MCQs demonstrated modest overall psychometric quality.
• LLM-generated items (mean difficulty 0.79, point-biserial 0.17, discrimination 0.08) were comparable to expert items (mean difficulty 0.81, point-biserial 0.19, discrimination 0.09).

Conclusions:

• Supervised fine-tuned LLMs can produce MCQs with psychometric quality similar to expert faculty.
• Automated item generation should supplement, not replace, manual MCQ development.
• Further research is needed for generalizability and optimizing LLM integration in assessment.