Using Pretrained Large Language Models for AI-Driven Assessment in Medical Education

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Ethics in Medicine

Background:

• Assessing medical students in competency-based programs is challenging due to time constraints and the need for objective feedback.
• Traditional assessment methods may lack consistency and targeted feedback, especially in complex subjects like moral injury.

Purpose of the Study:

• To pilot the use of large language models (LLMs) with retrieval-augmented generation for assessing student understanding of moral injury.
• To evaluate the efficiency, objectivity, and effectiveness of LLM-based assessment in a medical ethics course.

Main Methods:

• LLMs (Google Gemini 1.5 Pro, Google Gemini 2.0 Flash, OpenAI ChatGPT-4o) were used to generate a grading rubric from seminal articles on moral injury.
• 165 student responses were assessed by three LLMs using the generated rubric.
• LLM scoring accuracy was compared against two experienced educators to establish validity evidence.

Main Results:

• Google Gemini 1.5 Pro successfully generated a nuanced grading rubric for moral injury.
• OpenAI ChatGPT-4o achieved the highest interrater reliability (0.77 and 0.68) when compared to human reviewers, exceeding inter-reviewer agreement (0.57).

Conclusions:

• LLMs offer a promising tool for enhancing the efficiency and objectivity of medical student assessment, particularly in specialized ethical topics.
• Faculty oversight is crucial to ensure ethical accountability and mitigate potential biases in AI-driven assessments.