Automatic Coding of Short Text Responses via Clustering in Educational Assessment

Area of Science:

• Educational assessment
• Natural Language Processing (NLP)
• Computational linguistics

Background:

• Automatic coding of short text responses offers potential for large-scale educational assessments.
• Implementing NLP and statistical modeling requires robust software components and validated datasets.

Purpose of the Study:

• To implement and evaluate baseline NLP and statistical modeling methods for automatic coding of short text responses.
• To assess the accuracy and performance of the automatic coding system using Programme for International Student Assessment (PISA) 2012 data.
• To examine the impact of different methods, parameters, and sample sizes on system performance.

Main Methods:

• Utilized open-license software components for natural language processing and statistical modeling.
• Analyzed free text responses from 10 items within the PISA 2012 dataset (Germany).
• Evaluated system performance by comparing automatic codings against human codings.

Main Results:

• The automatic coding system demonstrated fair to excellent agreement with human codings (Kappa values ranging from 0.41 to 0.85).
• Items requiring specific semantic concept identification were coded accurately.
• Performance remained consistent with larger sample sizes and acceptable with smaller ones.

Conclusions:

• Automatic coding of short text responses is a viable and accurate method for educational assessment.
• The implemented system shows potential for innovation in large-scale testing and data analysis.
• Findings support the integration of NLP tools to enhance efficiency and consistency in educational evaluations.