Bias and Stability of Single Variable Classifiers for Feature Ranking and Selection

Area of Science:

• Machine Learning
• Data Science
• Computational Statistics

Background:

• Feature ranking is crucial for supervised dimension reduction, especially with high-dimensional data or limited computational resources.
• Single Variable Classifier (SVC) ranking offers a computationally efficient alternative to complex methods, assessing features based on individual classifier performance.

Purpose of the Study:

• To investigate the bias and stability of Single Variable Classifier (SVC) feature ranking methods.
• To determine the influence of classifier choice on SVC rankings versus the inherent discriminative power of features.
• To evaluate the benefits of using homogeneous versus heterogeneous classifiers in SVC and their impact on final classification performance.

Main Methods:

• Extensive empirical study of Single Variable Classifier (SVC) ranking.
• Comparison of feature rankings generated using homogeneous classifiers versus heterogeneous classifier ensembles.
• Calculation of prediction performance loss associated with using the same classifier for both SVC ranking and final classification.

Main Results:

• The choice of classifier significantly influences SVC rankings, indicating potential bias.
• The common practice of using the same classifier for feature ranking and final classification does not consistently yield optimal prediction performance.
• Heterogeneous classifier ensembles can provide more unbiased feature rankings and potentially enhance final classification accuracy.

Conclusions:

• SVC feature rankings are susceptible to classifier bias, challenging the assumption that classifier choice is secondary to feature discriminative power.
• Employing heterogeneous classifier ensembles in SVC offers a promising avenue for achieving more robust and unbiased feature rankings.
• Optimizing the selection of classifiers for both the ranking and classification stages is essential for maximizing predictive performance.