PaSTO-GNN: prompt-aware spatio-temporal graph neural networks for automatic essay scoring

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Educational Technology

Background:

• Automatic Essay Scoring (AES) seeks objective student writing assessment.
• Current deep learning models for AES often neglect essay discourse progression.
• The spatio-temporal nature of writing, where meaning evolves, is underutilized.

Purpose of the Study:

• To develop a novel deep learning framework for AES that incorporates spatio-temporal essay composition.
• To introduce a prompt-aware Spatio-Temporal Graph Neural Network (PaSTO-GNN) for enhanced essay evaluation.
• To improve the accuracy and objectivity of automated writing assessments.

Main Methods:

• Representing essays as spatio-temporal graphs with sentences as nodes.
• Utilizing RoBERTa (via LoRA) and Sentence-BERT (SBERT) for sentence embeddings, combined with prompt embeddings.
• Employing a spatio-temporal message passing network, BiGRU, temporal attention pooling, and CORAL-based ordinal prediction with prompt calibration.

Main Results:

• PaSTO-GNN achieved a Quadratic Weighted Kappa (QWK) of 0.8329 and Pearson correlation of 0.8168 on the AES 2.0 benchmark dataset.
• The model demonstrated superior performance by effectively integrating discourse progression and prompt-specific information.
• Prompt calibration significantly improved the alignment of predicted scores with human judgments.

Conclusions:

• Integrating spatio-temporal discourse modeling with prompt-aware representations significantly enhances AES performance.
• PaSTO-GNN offers a more comprehensive approach to automated essay evaluation by considering the evolving nature of text.
• The proposed method provides a promising direction for developing more accurate and reliable automated writing assessment tools.