Scoring Graphical Responses in TIMSS 2019 Using Artificial Neural Networks.

Area of Science:

• Artificial Intelligence
• Educational Measurement
• Computer Science

Background:

• Automated scoring of graphical student responses is not yet established for large-scale assessments.
• Traditional methods for scoring complex constructed-response items can be labor-intensive and prone to variability.

Purpose of the Study:

• To propose and compare artificial neural network approaches for automated scoring of image-based responses.
• To evaluate the accuracy of convolutional neural networks (CNNs) against feed-forward neural networks for classifying student drawings.

Main Methods:

• Utilized a TIMSS 2019 item with free drawing responses.
• Compared classification accuracy between convolutional neural networks (CNNs) and feed-forward neural networks.
• Implemented an item response theory-based method for selecting training data.

Main Results:

• CNNs demonstrated superior performance over feed-forward networks in both loss and accuracy.
• CNN models achieved up to 97.53% accuracy in classifying image responses, comparable to human raters.
• The most accurate CNN models correctly identified errors made by human raters.

Conclusions:

• CNN-based automated scoring is a highly accurate method for evaluating graphical student responses.
• This technology can potentially reduce costs and workload associated with human raters in international large-scale assessments.
• Automated scoring using CNNs can enhance the validity and comparability of scoring complex constructed-response items.