Evaluating Neural Networks Architectures for Competency Prediction from Process Data Using PISA Computer-Based...

Area of Science:

• Educational Measurement
• Computer Science

Background:

• Computer-based assessments generate valuable process data from examinee interactions.
• Analyzing this data can reveal insights into student proficiency and test performance.

Purpose of the Study:

• To apply recurrent neural networks (RNNs) to predict item-level correctness and assessment-level latent proficiency using process data.
• To examine the impact of features, architectural complexity, and variability on model performance.

Main Methods:

• Utilized process data from the U.S. PISA 2012 computer-based mathematics assessment sample.
• Applied various RNN architectures (standard RNN, GRU, LSTM) and analyzed expert-engineered features, architectural complexity, action variability, and score variability.

Main Results:

• Item-level models achieved good predictive performance (AUC ≈ 0.80).
• Moderate correlations were found between latent proficiency predictions and actual scores.
• Expert features improved item-level model efficiency and assessment-level model performance, especially with low action variability.

Conclusions:

• Simple neural network architectures are sufficient for modeling process data with limited action variability.
• Combining action sequences with expert-engineered features enhances accuracy, efficiency, and interpretability in educational assessments.