Evaluating stability and comparing output of feature selectors that optimize feature subset cardinality

Area of Science:

• Machine Learning
• Data Science
• Statistical Modeling

Background:

• Feature selection stability is critical for reliable machine learning systems.
• Evaluating the robustness of feature selection processes, especially those yielding variable-sized subsets, is an ongoing challenge.
• Existing methods for assessing feature selection stability are often limited or lack a unified framework.

Purpose of the Study:

• To introduce novel measures for evaluating the stability and similarity of feature selection methods.
• To provide a unifying framework for understanding and assessing feature selection stability.
• To demonstrate the utility of these measures in evaluating the performance and reliability of different feature selection approaches.

Main Methods:

• Development of several new feature selection stability measures.
• Adaptation of existing stability measures within a unified framework.
• Introduction of similarity measures to compare feature selection processes (e.g., different methods or parameter settings).
• Empirical evaluation of the proposed measures on various datasets and feature selection scenarios.

Main Results:

• The proposed measures offer broad insights into the stability of feature selection processes.
• Detailed analysis of the properties and information gained from the considered stability and similarity measures.
• Demonstration of how these measures can be used to quantitatively assess the reliability of feature selection methods.
• The similarity measures effectively compare outputs from different feature selection runs or methods.

Conclusions:

• The introduced stability and similarity measures provide a robust framework for evaluating feature selection methods.
• These measures enhance the reliability assessment of machine learning systems by quantifying feature selection robustness.
• The findings support informed selection and parameter tuning of feature selection algorithms for improved model performance.