Evaluating large language models for criterion-based grading from agreement to consistency

Area of Science:

• Artificial Intelligence
• Educational Technology
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in various applications.
• Automated grading systems are crucial for educational efficiency.
• The efficacy of LLMs in criterion-based grading requires thorough evaluation.

Purpose of the Study:

• To assess the capability of LLMs in criterion-based grading.
• To investigate the effect of detailed prompt engineering on grading performance.
• To understand the role of domain-specific knowledge in LLM grading.

Main Methods:

• Quantitative analysis comparing LLM performance against human benchmarks.
• Evaluation of LLMs using well-established grading criteria.
• Experimentation with prompt engineering techniques to refine LLM instructions.

Main Results:

• Free LLMs demonstrated proficiency in criterion-based grading.
• LLMs exhibited a nuanced understanding of grading criteria.
• Domain-specific understanding proved more critical than model complexity for accurate grading.

Conclusions:

• LLMs are capable of delivering criterion-based educational feedback.
• Prompt engineering significantly impacts LLM grading quality.
• LLMs offer a scalable solution for providing educational feedback.