ML-based validation of experimental randomization in learning games.

Area of Science:

• Experimental research methodology
• Computational statistics
• Data science in healthcare

Background:

• Randomization is crucial for experimental validity but can be compromised.
• Existing methods for randomization validation are limited.
• Machine learning (ML) offers potential for enhanced validation.

Purpose of the Study:

• To introduce and evaluate machine learning models as supplementary tools for validating participant randomization.
• To assess the performance of supervised and unsupervised ML models in detecting randomization patterns.
• To identify predictors of assignment bias using feature importance analysis.

Main Methods:

• Developed a learning direction game with dichotomized scenarios for participant assignment.
• Evaluated supervised ML models (logistic regression, decision tree, support vector machine) and unsupervised ML models (k-means, k-nearest neighbors, artificial neural networks).
• Utilized synthetic data augmentation to address sample size limitations and analyzed feature importance.

Main Results:

• Supervised ML models achieved a maximum accuracy of 87% in classifying randomized assignments, especially after synthetic data augmentation.
• Unsupervised models, including artificial neural networks (ANN), performed less effectively, with ANN exhibiting overfitting.
• Feature importance analysis successfully identified key predictors contributing to assignment bias.

Conclusions:

• Machine learning models, particularly supervised approaches, show promise for validating experimental randomization.
• The effectiveness of ML-based validation is contingent upon sample size and the complexity of the experimental design.
• Further research is needed to explore the applicability of this methodology across diverse experimental settings.