Beta Distribution-Based Cross-Entropy for Feature Selection.

Area of Science:

• Machine Learning
• Data Mining
• Computational Statistics

Background:

• High-dimensional data analysis presents challenges in machine learning and data mining.
• Feature selection is crucial for improving predictive accuracy and data interpretability.
• Conventional resampling methods for feature selection are computationally expensive and can be over-optimistic.

Purpose of the Study:

• To propose a novel cross-entropy method based on beta distribution for feature selection.
• To enhance the precise estimation of generalization ability in feature selection.
• To develop a computationally efficient and robust feature selection framework.

Main Methods:

• Developed Beta distribution-based cross-entropy (BetaDCE) for feature selection.
• Estimated probability density using beta distribution and computed cross-entropy via expected value.
• Analyzed generalization ability, bias-variance trade-off, and robustness through empirical experiments.

Main Results:

• BetaDCE demonstrated a favorable bias-variance trade-off for generalization ability.
• On an exclusive or-like (XOR-like) dataset, BetaDCE achieved a significantly lower false discovery rate.
• For the leukemia dataset, BetaDCE yielded an AUC of 0.93 with four features, outperforming the original method (AUC 0.83 with 50 features).
• The metabonomic dataset showed significantly higher AUC for predictions using features selected by BetaDCE.

Conclusions:

• Beta distribution-based cross-entropy (BetaDCE) offers a precise and efficient approach to feature selection.
• BetaDCE effectively identifies irrelevant and redundant features, leading to improved predictive performance with fewer features.
• The proposed method serves as a general and efficient framework for feature selection in high-dimensional data analysis.