Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of...

Area of Science:

• Natural Language Processing
• Educational Technology
• Computational Linguistics

Background:

• Automated essay scoring (AES) research increasingly utilizes hybrid models combining feature engineering and deep neural networks (DNNs).
• Existing state-of-the-art findings primarily focus on holistic scoring tasks, leaving trait-specific scoring less explored.
• Trait scores (e.g., content, organization, language quality) offer more granular feedback than holistic scores.

Purpose of the Study:

• To compare the effectiveness of feature-based, embedding-based, and hybrid models for trait-specific AES.
• To investigate the interplay between linguistic features and deep learning embeddings in essay scoring.
• To analyze performance across different essay traits (content, organization, language) and learner groups (L1, L2).

Main Methods:

• Trained trait-specific models using four prompts from L1 and L2 learner essay corpora.
• Compared three model variants: linguistic features (220 features), DistilBERT embeddings, and a hybrid approach.
• Conducted addition and ablation tests to analyze feature interactions for specific traits.

Main Results:

• Feature-based models slightly outperformed embedding-based models when trained on single resources, especially for organization traits.
• Hybrid models consistently outperformed single-resource models, indicating complementary information from features and embeddings.
• Specific features (morphological complexity, length, lexical complexity, error/occurrence) enhanced embedding-based models for particular traits.

Conclusions:

• Hybrid models integrating linguistic features and deep learning embeddings are effective for trait-specific AES.
• Linguistic features and embeddings capture distinct, complementary aspects of essay quality.
• Understanding feature interactions can further optimize AES models for targeted trait assessment.