Investigating response behavior through TF-IDF and Word2vec text analysis: A case study of PISA 2012 problem-solving...

Area of Science:

• Educational measurement and assessment
• Computational linguistics
• Machine learning in education

Background:

• Process data in computer-based problem-solving (CBPS) evaluation contains implicit information.
• Irregular data structures hinder effective feature extraction, causing information loss.
• Process-response behavior shares similarities with textual data, suggesting text analysis applicability.

Purpose of the Study:

• Investigate Term Frequency-Inverse Document Frequency (TF-IDF) and Word to Vector (Word2vec) for extracting response behavior features.
• Compare supervised and unsupervised machine learning methods using these extracted features.
• Evaluate the predictive, analytical, and clustering performance of these approaches.

Main Methods:

• Applied TF-IDF and Word2vec to extract features from the PISA 2012 CBPS dataset.
• Utilized classical machine learning algorithms: Random Forest, Support Vector Machine (SVM), Logistic Regression, k-Nearest Neighbors (k-NN).
• Employed k-means for unsupervised clustering of response behavior patterns.

Main Results:

• TF-IDF effectively extracted key response behaviors; Word2vec captured features from sequenced behaviors.
• In supervised learning, TF-IDF-based Random Forest performed best; SVM based on Word2vec was second.
• Word2vec models generally outperformed TF-IDF models in F1-score, accuracy, and recall across algorithms.
• K-means clustering effectively identified distinct response behavior patterns.

Conclusions:

• Text analysis methods (TF-IDF, Word2vec) are transferable to educational and psychological assessment.
• These methods yield rich feature representations for fine-grained assessment.
• Findings support personalized learning and offer novel insights for educational assessment research and practice.