Computerization of Off-Topic Essay Detection: A possibility?

Area of Science:

• Natural Language Processing
• Machine Learning
• Educational Technology

Background:

• Student essay evaluation is time-consuming, requiring assessment of semantic content beyond grammar.
• Automated off-topic essay detection is crucial for online learning platforms, especially post-COVID-19.
• Evaluating Pakistani English essays for topic relevance presents unique challenges.

Purpose of the Study:

• To investigate the feasibility of fully automating off-topic essay detection using machine learning.
• To compare various embedding techniques for measuring essay-prompt similarity.
• To determine the optimal combination of embeddings and classifiers for accurate off-topic detection.

Main Methods:

• Exploration and implementation of recent embedding techniques (e.g., Word Mover Distance, average word embeddings, IDF weighted word embeddings).
• Comparison of embedding technique performance across 10 benchmark datasets.
• Evaluation using 6 different machine learning classifiers.

Main Results:

• The combination of Word Mover Distance, average word embeddings, and IDF weighted word embeddings demonstrated superior performance.
• Random Forest classifier, when used with the combined embeddings, yielded the highest accuracy.
• An overall accuracy of 93.5% was achieved for off-topic essay detection.

Conclusions:

• Fully automated off-topic essay detection is achievable with current machine learning techniques.
• The proposed method offers a reliable and efficient solution for educational platforms.
• The optimal strategy involves a specific ensemble of word embeddings and the Random Forest classifier.