Large language models for generating key-feature questions in medical education

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Cardiology Training

Background:

• Current methods for developing Key Feature Questions (KFQs) can be time-consuming.
• The Medical Council of Canada provides guidelines for KFQ development.
• Large Language Models (LLMs) show potential for automating content generation.

Purpose of the Study:

• To evaluate the quality of KFQs generated by OpenAI's o3 model.
• To develop a standardized prompt and evaluation metric for AI-generated KFQs.
• To assess the alignment of AI-generated KFQs with established development guidelines.

Main Methods:

• A generic prompt aligned with the Medical Council of Canada's KFQ guidelines was developed.
• Twenty cardiology-focused KFQs were generated using OpenAI's o3 model and ESC guidelines.
• Two cardiology experts assessed KFQs using a quality checklist; a third resolved disagreements.

Main Results:

• 100% of generated KFQs were rated 'Acceptable' (15% 'Accept as is', 85% 'Accept with minor revisions').
• Overall checklist compliance reached 93.7%, with high scores in key feature definition and scenario plausibility.
• Areas for improvement included 'killer' response inclusion (50%) and distractor plausibility (77.8%).

Conclusions:

• LLM-generated KFQs, guided by structured prompts, meet high-quality standards with minor revisions.
• AI-assisted workflows can streamline KFQ development for Competency-Based Medical Education (CBME).
• Expert review remains crucial for clinical accuracy and patient safety in AI-assisted medical education.