Ensemble Estimation of Information Divergence ^†

Area of Science:

• Information Theory
• Statistical Inference
• Machine Learning

Background:

• Nonparametric estimation of information divergence functionals is crucial for analyzing relationships between random variables.
• Existing methods often require restrictive assumptions about density support sets or complex boundary calculations.
• A need exists for robust estimators that perform well across diverse support set conditions.

Purpose of the Study:

• To develop a nonparametric divergence estimator that does not require prior knowledge of the support set boundary.
• To generalize ensemble estimation theory for improved divergence estimation rates.
• To propose an empirical estimator for Rényi-α divergence with enhanced performance.

Main Methods:

• Derivation of mean squared error (MSE) convergence rates for a leave-one-out kernel density plug-in estimator.
• Generalization of optimally weighted ensemble estimation theory.
• Development of an empirical estimator for Rényi-α divergence.

Main Results:

• The proposed estimator achieves improved MSE convergence rates for general bounded density support sets without boundary knowledge.
• A new divergence estimator achieves the parametric rate for sufficiently smooth densities.
• The empirical Rényi-α divergence estimator demonstrates superior performance and robustness to tuning parameters compared to standard methods.

Conclusions:

• The developed methods provide a more flexible and accurate approach to estimating information divergence functionals.
• The proposed Rényi-α divergence estimator offers significant advantages in mean squared error, particularly in high-dimensional settings.
• The estimator's robustness and performance are validated through simulations and application to Bayes error rate estimation.