Feature selection by optimizing a lower bound of conditional mutual information.

Area of Science:

• Machine Learning
• Information Theory
• Data Science

Background:

• Feature selection is crucial for improving model performance and interpretability in high-dimensional data.
• Existing information theory-based methods often rely on strong assumptions, limiting their applicability.
• Distinguishing between redundant and irrelevant features remains a challenge in robust feature selection.

Purpose of the Study:

• To propose a unified framework for feature selection based on conditional mutual information (CMI).
• To develop a novel feature selection algorithm optimizing a lower bound of CMI under weaker assumptions.
• To introduce a new metric for evaluating feature selection methods using simulated data.

Main Methods:

• Developed a unified framework for approximating high-dimensional CMI.
• Proposed a new feature selection algorithm optimizing a CMI lower bound with a plug-in component.
• Introduced a novel evaluation metric for feature selection methods based on simulated data.

Main Results:

• The proposed method successfully rederived state-of-the-art algorithms within the unified framework.
• The new algorithm demonstrated improved robustness in distinguishing redundant from irrelevant features.
• Comparative experiments showed promising performance against existing methods using the new metric and classifier accuracy.

Conclusions:

• The unified framework provides a flexible approach for information theory-based feature selection.
• The novel algorithm offers a robust and effective solution for feature selection problems.
• The proposed evaluation metric facilitates better assessment of feature selection method performance.