Use of Large Language Models for Rapid Quantitative Feedback in Case-Based Learning: A Pilot Study

Area of Science:

• Medical education technology
• Artificial intelligence in healthcare

Background:

• Interactive case-based learning is valuable for medical students.
• Automated scoring of student interactions can enhance feedback.
• Large language models (LLMs) offer potential for educational assessment.

Purpose of the Study:

• To evaluate the efficacy of GPT-4o in scoring medical student interactions with virtual cases.
• To assess the correlation between LLM-generated scores and expert evaluations.
• To determine if calibration can improve LLM scoring accuracy.

Main Methods:

• Medical students interacted with virtual cases presented by an LLM.
• GPT-4o scored student interactions.
• LLM scores were compared against expert human scorers.
• Calibration methods were applied to address discrepancies.

Main Results:

• GPT-4o demonstrated a high correlation with expert scorer evaluations.
• A notable agreement was observed between LLM and expert scoring.
• Calibration effectively reduced the difference between LLM and expert scores.

Conclusions:

• LLMs, specifically GPT-4o, show significant promise for evaluating medical student performance in case-based scenarios.
• Automated scoring by LLMs can be a reliable adjunct to traditional expert assessment.
• Calibration is a viable strategy to enhance the accuracy of LLM-based educational scoring.