Comparing machine learning and artificial neural network models in psychological research: a ROC-based analysis

Area of Science:

• Psychological Assessment
• Machine Learning
• Artificial Intelligence

Background:

• Data-driven methods are increasingly used in psychological assessment.
• The comparative performance of artificial neural networks versus traditional machine learning models in selection contexts is not well-established.
• Limited comparative evidence exists using real-world psychological datasets and ROC-based evaluation.

Purpose of the Study:

• To compare the effectiveness of traditional machine learning models against artificial neural networks in applied psychological selection.
• To evaluate model performance using accuracy and receiver operating characteristic (ROC) analysis on a real-world dataset.

Main Methods:

• Compared logistic regression, decision tree, and random forest models with a feedforward artificial neural network.
• Utilized a dataset of 4,155 university entrance examination applicants.
• Evaluated models using accuracy and Area Under the Curve (AUC) from ROC analysis.

Main Results:

• Logistic regression yielded the highest predictive performance (accuracy = 0.973, AUC = 0.99), closely followed by random forest (accuracy = 0.961, AUC = 0.98).
• The artificial neural network achieved lower discriminative ability (AUC = 0.87) and showed signs of overfitting.
• Biology, chemistry, and numerical reasoning were identified as key predictors.

Conclusions:

• Traditional machine learning models offer more stable, interpretable, and robust performance for medium-sized, structured psychological datasets compared to shallow neural networks.
• Model selection and inductive bias are crucial in applied psychological research.
• Classical machine learning approaches remain valuable for selection and assessment contexts.