A refined approach for evaluating small datasets via binary classification using machine learning

Area of Science:

• Machine Learning
• Statistical Analysis
• Educational Research

Background:

• Classical statistical methods are often complemented or replaced by machine learning (ML).
• Small datasets, common in fields like education research, pose challenges related to bias and spurious findings.
• Evaluating ML performance on limited data requires specialized techniques to ensure reliability.

Purpose of the Study:

• To present a refined methodology for evaluating binary classification performance using ML on small datasets.
• To address issues of bias and chance in ML model evaluation within data-limited research contexts.
• To provide guidelines for selecting appropriate evaluation metrics for small-dataset ML applications.

Main Methods:

• Implementation of a non-parametric permutation test to assess the generalizability of ML model results.
• Utilization of repeated nested cross-validation for bias-free and reliable performance estimation.
• Comparative analysis of various evaluation metrics, including the Matthews correlation coefficient.

Main Results:

• Repeated nested cross-validation demonstrates minimal bias and high reliability, with results largely independent of chance.
• The permutation test effectively quantifies the probability of results generalizing to new, unseen data.
• The Matthews correlation coefficient is identified as a robust metric for binary classification when classes have equal importance, showing low bias and chance of coincidental success.

Conclusions:

• A combination of evaluation metrics is recommended for training and assessing ML classifiers to leverage their respective strengths.
• The proposed approach, incorporating permutation tests and nested cross-validation, is crucial for accurate ML analysis of small datasets.
• Avoiding biases is paramount when applying machine learning techniques to small datasets, particularly in sensitive research areas like education.