Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Artificial Intelligence (AI)

Background:

• Large Language Models (LLMs) are increasingly utilized in Automated Essay Scoring (AES) due to advancements in AI.
• LLMs offer potential for efficient and unbiased writing assessment.
• The reliability and alignment of LLMs with human raters in AES require thorough investigation.

Purpose of the Study:

• To assess the reliability of LLMs in Automated Essay Scoring (AES).
• To explore the impact of prompt engineering, temperature settings, and multi-level rating dimensions on LLM scoring performance.
• To evaluate the alignment of LLM scores with human evaluations.

Main Methods:

• Investigated the effect of prompt engineering strategies (criteria and sample-referenced justification) on LLM performance.
• Analyzed the influence of temperature settings on LLM output consistency.
• Evaluated LLM performance across multiple writing dimensions (Ideas, Organization) using Quadratic Weighted Kappa (QWK).

Main Results:

• Prompt engineering significantly improved LLM reliability, with GPT-4 outperforming GPT-3.5 and Claude 2.
• Lower temperature settings resulted in LLM scores more consistent with human evaluations.
• GPT-4 demonstrated strong performance in 'Ideas' (QWK=0.551) and 'Organization' (QWK=0.584) with optimized prompts.

Conclusions:

• LLMs, particularly GPT-4, show significant potential for reliable and accurate Automated Essay Scoring.
• Careful prompt engineering and temperature control are crucial for maximizing LLM performance and fairness in AES.
• Findings suggest LLMs can enhance writing instruction and feedback in the AI-driven educational landscape.