Evaluating feature selection strategies for high dimensional, small sample size datasets.

Area of Science:

• Biomedical Informatics
• Bioinformatics
• Computational Biology

Background:

• High-dimensional (HD) biomedical datasets, common in genomics, often present a small sample size (SSS).
• Traditional evaluation of feature selection (FS) schemes using classifier accuracy may be insufficient for HD/SSS data due to the curse of dimensionality and potential lack of representativeness.
• Robustness, defined as invariance to training data changes, is a critical but often overlooked metric for FS schemes in HD/SSS contexts.

Purpose of the Study:

• To analyze and evaluate different strategies for comparing FS schemes on HD/SSS biomedical datasets.
• To introduce and utilize a new performance measure, Robustness, for FS schemes.
• To compare the efficacy of five distinct FS schemes on diverse HD/SSS gene and protein expression datasets.

Main Methods:

• Quantitative comparison of five FS schemes: T-test, F-test, Kolmogorov-Smirnov Test, Wilks Lambda Test, and Wilcoxon Rank Sum Test.
• Evaluation using classifier accuracy (K-Nearest Neighbor and Random Forest) and the newly defined Robustness measure.
• Application to five HD/SSS biomedical datasets from cancer, bone lesions, celiac disease, and coronary heart disease studies.

Main Results:

• The Wilcoxon Rank Sum Test significantly outperformed the other four FS schemes in both classification accuracy and robustness.
• Demonstrated that classifier accuracy alone is not always sufficient for evaluating FS schemes on HD/SSS datasets.
• Highlighted the importance of the Robustness metric in assessing FS scheme reliability for HD/SSS data.

Conclusions:

• Both classifier accuracy and Robustness are essential considerations when selecting an appropriate FS scheme for HD/SSS biomedical datasets.
• The Wilcoxon Rank Sum Test emerges as a highly effective FS scheme for HD/SSS biomedical data.
• Future evaluations of FS schemes for HD/SSS data should incorporate Robustness as a key performance indicator.