Automated post scoring: evaluating posts with topics and quoted posts in online forum.

Area of Science:

• Educational Technology
• Natural Language Processing
• Artificial Intelligence in Education

Background:

• Manual evaluation of online forum posts is time-consuming for instructors.
• Existing automated text assessment tools do not fully capture the nuances of discussion forum evaluations.
• There is a need for automated grading that considers content relevance and supporting evidence within discussion threads.

Purpose of the Study:

• To introduce a novel task, Automated Post Scoring (APS), for grading online discussion posts.
• To develop a model that evaluates both writing quality and topic relevance of student posts.
• To enhance post evaluation by incorporating information from quoted posts and discussion topics.

Main Methods:

• A mixture model was proposed, integrating a hierarchical text model for writing quality, a semantic matching model for topic relevance, and a semantic representation model for integrating quoted posts.
• A new dataset, the Online Discussion Dataset, was constructed, comprising 2,542 posts from 694 students.
• Evaluation metrics included correlation and residual-based measures to assess model performance.

Main Results:

• The proposed Automated Post Scoring (APS) models demonstrated effectiveness in evaluating online discussion posts.
• Incorporating topics and quoted posts significantly improved APS performance compared to evaluating posts in isolation.
• Performance gains exceeded 9 percent on the Quadratic Weighted Kappa (QWK) metric.

Conclusions:

• Automated Post Scoring (APS) provides a viable and effective method for automating the grading of online discussion forums.
• The integration of topic relevance and supporting arguments from quoted posts is crucial for accurate automated evaluation.
• This approach can substantially reduce instructor workload while maintaining or improving evaluation quality.