Analysis of Information-Based Nonparametric Variable Selection Criteria.

Area of Science:

• Machine Learning
• Statistical Modeling
• Data Science

Background:

• Nonparametric Generative Tree Models are complex statistical tools.
• Selecting relevant predictors is crucial for model accuracy and interpretability.
• Information-theoretic criteria are commonly used for feature selection.

Purpose of the Study:

• To analyze the behavior of Conditional Infomax Feature Extraction (CIFE) and Joint Mutual Information (JMI) in Generative Tree Models.
• To compare these criteria against Conditional Mutual Information (CMI).
• To derive explicit formulas for these criteria within the generative tree model context.

Main Methods:

• Investigated two information-based feature selection criteria: CIFE and JMI.
• Analyzed their derivation as approximations of Conditional Mutual Information (CMI).
• Derived explicit mathematical formulas for CMI and its approximations in generative tree models.

Main Results:

• Demonstrated that CIFE and JMI can yield different predictor selection orders compared to CMI.
• Obtained explicit analytical expressions for CMI and its approximations.
• Derived byproduct expressions for the entropy of multivariate Gaussian mixtures and their mutual information.

Conclusions:

• CIFE and JMI are not always interchangeable with CMI for feature selection in Generative Tree Models.
• The derived formulas provide theoretical insights into these information-theoretic criteria.
• The findings contribute to a better understanding of variable selection in complex generative models.