Performance evaluation of automated scoring for the descriptive similarity response task

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Education
• Psychometrics

Background:

• The Semantic Similarity Test (SST) assesses understanding of conceptual relationships.
• Automated scoring systems are increasingly used in educational and psychological assessments.
• Evaluating the performance of machine learning models against human judgment is crucial for their adoption.

Purpose of the Study:

• To investigate the efficacy of a machine learning model in replicating human scoring on the Japanese SST.
• To compare the accuracy and reliability of automated scoring with human performance.

Main Methods:

• A bidirectional encoder representations from transformer (BERT) model was trained using participant responses and classification criteria from the SST.
• Fivefold cross-validation was employed for model evaluation.
• Model performance was assessed using accuracy and inter-rater reliability metrics.

Main Results:

• The model achieved 83% accuracy when trained on both responses and classification criteria (Experiment 1).
• In a separate test set, the model scored 80% accuracy (Experiment 2).
• Human-model inter-rater reliability (0.63) closely matched human-human reliability (0.67-0.72).

Conclusions:

• Machine learning models can achieve human-level performance in scoring the Japanese SST.
• Automated scoring systems offer a reliable alternative to human scoring for similarity tasks.
• This research supports the use of AI for objective and efficient assessment in language and cognitive studies.