Assessing the use of multiple sources in student essays

Area of Science:

• Natural Language Processing
• Educational Technology
• Computational Linguistics

Background:

• Automated essay scoring (AES) is crucial for large-scale educational assessments.
• Evaluating student comprehension of multiple source texts presents unique challenges for AES systems.
• Existing AES methods may struggle to capture both explicit information and inferential reasoning.

Purpose of the Study:

• To investigate and compare the effectiveness of different automated approaches for scoring student essays based on multiple texts.
• To determine the suitability of pattern-matching, latent semantic analysis (LSA), and machine learning for identifying explicit content and inferences.
• To inform the development of more robust AES systems for complex writing tasks.

Main Methods:

• Developed and evaluated a "multi-word" pattern-matching approach for flexible word and phrase identification.
• Applied latent semantic analysis (LSA) to compare essay content with source texts using vector representations.
• Utilized support vector machines (SVMs), a machine learning technique, for text classification based on a training corpus.

Main Results:

• The LSA-based system demonstrated superior performance in detecting explicit content directly from source texts.
• The multi-word pattern-matching approach proved more effective at identifying student inferences that extended beyond or across texts.
• No single method was universally superior, indicating the need for hybrid approaches.

Conclusions:

• Combining multiple natural language processing (NLP) techniques offers the most promising strategy for comprehensive automated essay analysis.
• Future AES systems should integrate diverse methods to accurately assess both factual recall and inferential understanding.
• The findings highlight the complexity of evaluating higher-order thinking skills in student writing through automated means.