Ensemble estimators for multivariate entropy estimation.

Area of Science:

• Statistics
• Information Theory
• Machine Learning

Background:

• Estimation of density functionals like entropy and mutual information is crucial in statistics and information theory.
• Many existing estimators suffer from the curse of dimensionality, leading to slow convergence rates (e.g., O(T^-γ)).
• Common examples include kernel density estimators and k-nearest neighbor (k-NN) methods.

Purpose of the Study:

• To develop a new estimation method that overcomes the curse of dimensionality for density functionals.
• To achieve a faster, dimension-invariant convergence rate for improved accuracy.
• To demonstrate the practical utility of the proposed method in key applications.

Main Methods:

• Propose a weighted affine combination of an ensemble of existing density functional estimators.
• Determine optimal weights by solving a convex optimization problem, feasible offline without training data.
• Utilize standard statistical estimation techniques and convex optimization.

Main Results:

• The proposed weighted estimator converges at a significantly faster, dimension-invariant rate of O(T^-1).
• Optimal weights can be determined efficiently via convex optimization.
• Demonstrated superior performance in estimating the Panter-Dite distortion-rate factor and Shannon entropy.

Conclusions:

• The weighted ensemble approach effectively mitigates the curse of dimensionality in density functional estimation.
• The method offers a computationally efficient and data-independent way to determine optimal weights.
• This technique provides a robust and accurate solution for critical applications in statistical inference and information theory.