Scale abbreviation with supervised machine learning: A comparison of feature selection techniques

Scale abbreviation is a crucial task for researchers aiming to reduce response burden and optimize data collection when using self-report instruments such as online surveys and questionnaires. Among various data-driven strategies available for scale abbreviation, supervised machine learning (SML) algorithms have emerged as a prominent approach due to their accuracy in predicting total scores from the original instrument. However, previous studies offer limited insights into how SML-abbreviated scales can be evaluated using both SML and psychometric metrics across different feature selection techniques. To address this gap, the current study aims to evaluate the effectiveness of seven feature selection methods: item-total-correlation-based filters (ITC), Minimum-Redundancy-Maximum-Relevance (MRMR), Lasso, Sequential Forward Selection (SFS), Sequential Backward Selection (SBS), Genetic Algorithms (GA), and Non-dominated Sorting Genetic Algorithms-II (NSGA-II), all used in conjunction with SML. Additionally, the psychometric properties of these SML methods are compared with two non-SML approaches. Using simulated datasets varying in sample size, model error, and factorial correlations, the study examines predictive accuracy, reliability, and the ability to recover both inter-subscale correlations and external criterion correlations. The findings indicate that no single method consistently excels across all conditions, with specific feature selection techniques performing better under certain circumstances. Key insights are provided to guide researchers in selecting appropriate feature selection methods based on their specific dataset characteristics and goals.