Relevance, redundancy, and complementarity trade-off (RRCT): A principled, generic, robust feature-selection tool

Area of Science:

• Machine Learning
• Data Mining
• Statistical Learning

Background:

• Feature selection (FS) is crucial for building efficient and accurate predictive models.
• Existing FS methods often struggle to balance feature relevance, redundancy, and complementarity.
• A unified framework addressing these three components is needed for robust feature selection.

Purpose of the Study:

• To introduce a novel heuristic feature selection algorithm named Relevance, Redundancy, and Complementarity Trade-off (RRCT).
• To integrate the key feature selection components—relevance, redundancy, and complementarity—into a principled algorithmic framework.
• To evaluate the performance of RRCT against existing FS algorithms in challenging settings.

Main Methods:

• The RRCT algorithm quantifies feature relevance and redundancy using information-theoretic transformations of rank correlation coefficients.
• Feature complementarity is measured using partial correlation coefficients.
• Empirical benchmarking involved comparing RRCT against 19 other FS algorithms on synthetic and real-world datasets, evaluating feature set matching and out-of-sample classification performance using Random Forests.

Main Results:

• RRCT demonstrates highly competitive performance in matching true feature sets across various datasets.
• The algorithm also shows strong out-of-sample performance in both binary and multi-class classification tasks.
• RRCT's effectiveness was observed in challenging settings, highlighting its robustness.

Conclusions:

• The proposed RRCT algorithm offers a principled and effective approach to feature selection by considering relevance, redundancy, and complementarity.
• RRCT is a strong contender among existing feature selection methods, particularly for classification tasks.
• Further investigation into the generalizability and optimal application scenarios for RRCT and other top-performing FS algorithms is warranted.