Utility of large language models for creating clinical assessment items

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Assessment Design

Background:

• Traditional methods for creating clinical and professional skills assessment (CPSA) items are labor-intensive.
• Evaluating the impact of emerging AI technologies like generative pretrained transformer (GPT) on assessment quality and cost is crucial.

Purpose of the Study:

• To compare student performance, examiner perceptions, and costs of GPT-assisted CPSA items versus standard methods.
• To assess the feasibility and effectiveness of using GPT for developing medical education assessment tools.

Main Methods:

• A prospective, controlled, double-blinded comparison of GPT-assisted and standard CPSA items.
• Two sets of six practical cases were developed for final-year medical students' formative assessment.
• Students were randomly assigned to assessment sets containing either GPT-assisted or standard cases.

Main Results:

• No statistically significant differences in item difficulty or discriminative ability were found between GPT-assisted and standard items.
• Examiners unanimously (100%) found GPT-assisted cases to be appropriately difficult and realistic.
• GPT-assistance yielded significant labor cost savings, averaging a 57% reduction per case.

Conclusions:

• GPT-assisted methods can produce high-quality CPSA items more cost-effectively than traditional approaches.
• Further research should explore GPT's generalizability in creating CPSA materials across diverse clinical domains.