Artificial intelligence assisted automated short answer question scoring tool shows high correlation with human...

Area of Science:

• Medical Education
• Artificial Intelligence in Education
• Natural Language Processing

Background:

• Optimizing short answer question (SAQ) assessment and feedback for medical undergraduates is challenging due to increasing student numbers and staff shortages.
• Developing scalable and effective assessment tools is crucial for medical education.
• Automated grading systems offer a potential solution to these challenges.

Purpose of the Study:

• To develop and evaluate an automated SAQ scoring tool (ASST) utilizing artificial intelligence (AI).
• To assess the feasibility of using large language models (LLMs) for grading SAQs based on provided rubrics.
• To provide personalized, granular feedback to students on their written answers.

Main Methods:

• Investigated the use of GPT-4, a large language model (LLM), for automated SAQ scoring in a Systematic Pharmacology course.
• LLM analyzed student answers against provided rubrics, extracting key information, scoring, and generating feedback.
• Evaluated LLM performance by averaging five sampled runs and compared scores against human examiner assessments for correlation.

Main Results:

• The automated SAQ scoring tool (ASST) demonstrated a high correlation with human examiner markings, with correlation coefficients of 0.93 and 0.96 across 30 student answers.
• Excellent inter-rater reliability was observed, with an intra-class correlation coefficient of 0.94 between the LLM and human graders.
• The AI-assisted tool provided scores that closely matched expert human judgment.

Conclusions:

• AI-assisted automated SAQ scoring tools show significant promise for transparent and flexible grading in medical education.
• The system's high correlation with human graders suggests its potential to reduce instructor workload.
• This approach offers students more granular and actionable feedback, enhancing the learning process.