Nonparametric estimation of Küllback-Leibler divergence

Area of Science:

• Information Theory
• Statistical Inference
• Machine Learning

Background:

• Kullback-Leibler divergence is a fundamental measure in information theory.
• Estimating KL divergence from finite samples is crucial for various statistical and machine learning applications.
• Standard plug-in estimators for KL divergence suffer from significant bias issues, particularly with rare events.

Purpose of the Study:

• To introduce a novel, bias-corrected estimator for Kullback-Leibler divergence.
• To analyze the theoretical properties of the new estimator, including bias, consistency, and asymptotic normality.
• To compare the performance of the new estimator against the standard plug-in estimator.

Main Methods:

• Development of a new estimator for KL divergence based on two independent samples.
• Theoretical analysis of the estimator's bias, proving exponential decay on finite alphabets.
• Mathematical derivation of consistency and asymptotic normality for the proposed estimator.
• Comparative analysis of the standard plug-in estimator, highlighting its infinite bias and slow convergence.

Main Results:

• The proposed estimator demonstrates exponentially decaying bias, outperforming the standard plug-in estimator.
• The new estimator is proven to be consistent and asymptotically normal.
• The standard plug-in estimator, while consistent and asymptotically normal, possesses an infinite bias.
• Modifications to the plug-in estimator to mitigate bias still result in slow bias decay (≤ O(1/n)).
• The results are extended to the estimation of symmetrized Kullback-Leibler divergence.

Conclusions:

• The newly introduced estimator provides a theoretically sound and practically superior method for estimating Kullback-Leibler divergence.
• The findings offer significant improvements over existing methods, especially in scenarios with limited or sparse data.
• Simulation results confirm the practical efficacy and asymptotic properties of the proposed estimator, even for small sample sizes.