Nonparametric k-nearest-neighbor entropy estimator.

Area of Science:

• Information Theory
• Statistical Inference
• Machine Learning

Background:

• Classical entropy estimators like Kozachenko-Leonenko have limitations with high-dimensional data and correlated variables.
• Nonuniform probability densities in local regions are often overlooked by traditional methods.
• Accurate entropy estimation is crucial for various statistical and machine learning applications.

Purpose of the Study:

• To introduce a novel nonparametric k-nearest-neighbor (k-NN) based entropy estimator.
• To enhance classical entropy estimation by accounting for nonuniform probability densities near sample points.
• To address limitations of existing estimators in high dimensions, correlated variables, and varying marginal variances.

Main Methods:

• Developed a k-nearest-neighbor (k-NN) based entropy estimation technique.
• Incorporated nonuniform probability density considerations within the k-NN neighborhood.
• Analyzed error heuristics for both proposed and classical estimators.
• Empirically tested the estimator across diverse distributions and increasing dimensions.

Main Results:

• The proposed k-NN entropy estimator demonstrated significant performance improvements compared to classical methods.
• Effectiveness was validated in scenarios with high dimensionality and near-functional relationships.
• The estimator showed robustness even when marginal variances of random variable components varied significantly.

Conclusions:

• The novel k-NN entropy estimator provides a more accurate and robust alternative to classical methods.
• This approach effectively handles complex data structures, including high dimensionality and variable correlations.
• The findings suggest broader applicability in statistical modeling and machine learning where precise entropy calculation is vital.