Can Large Language Models Replicate Systematic Review Outcome Classifications in Medical Education? A Pilot Study...

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Systematic Review Methodology

Background:

• Systematic reviews in medical education commonly use the Kirkpatrick framework for outcome classification.
• Manual coding of outcomes is labor-intensive and prone to subjectivity.
• Advancements in AI offer potential solutions for automating such tasks.

Purpose of the Study:

• To evaluate the performance of ChatGPT (GPT-5, August 2025 release) in classifying outcomes within medical education systematic reviews.
• To assess the agreement between AI-driven classification and human expert coding using the Kirkpatrick framework.

Main Methods:

• A proof-of-concept study was performed using 32 full-text articles from a systematic review on sepsis education.
• ChatGPT was employed to code outcomes based on the Kirkpatrick framework.
• Agreement was quantified using percentage agreement, unweighted kappa, and weighted kappa statistics.

Main Results:

• The agreement between ChatGPT and human coders was modest, with 50% percent agreement.
• Unweighted kappa (κ) was 0.170 (95% CI 0.000-0.458) and weighted kappa (κ) was 0.351 (95% CI 0.074-0.629).
• Most discrepancies occurred between adjacent levels of the Kirkpatrick framework.

Conclusions:

• ChatGPT demonstrates potential for assisting in the classification of outcomes in medical education systematic reviews.
• Current performance indicates limitations, with a need for refinement to improve accuracy and consistency.
• Further development and validation are necessary before widespread adoption of AI tools for systematic review data extraction.