QUEST-AI: A System for Question Generation, Verification, and Refinement using AI for USMLE-Style Exams

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Assessment and Evaluation

Background:

• Creating United States Medical Licensing Examination (USMLE) questions is time-consuming and expensive.
• Large Language Models (LLMs) show promise in answering medical questions but their generative potential is underexplored.

Purpose of the Study:

• To introduce QUEST-AI, a novel system utilizing LLMs for generating, identifying, and correcting USMLE-style medical exam questions.
• To evaluate the validity and utility of LLM-generated medical exam content.

Main Methods:

• Developed QUEST-AI, an LLM-based system for medical question generation and error correction.
• Created a test set of 50 LLM-generated and 50 human-generated questions.
• Conducted a two-part assessment with physicians and medical students to evaluate question validity and distinguishability.

Main Results:

• A majority of QUEST-AI-generated questions were deemed valid by a panel of clinicians.
• Strong correlations were observed between performance on LLM-generated and human-generated questions.
• Assessors had difficulty distinguishing between AI and human-generated questions.

Conclusions:

• QUEST-AI demonstrates a pioneering application of LLMs in medical education for efficient exam content development.
• The system offers a potentially cost-effective and accessible alternative for creating USMLE-style assessment materials.
• LLM-driven question generation can significantly enhance the ease and efficiency of preparing medical examination content.