Low-probability states, data statistics, and entropy estimation.

Area of Science:

• Complex Systems Analysis
• Information Theory
• Statistical Modeling

Background:

• Estimating the entropy of probability distributions in complex systems is crucial but difficult.
• Maximum likelihood estimators are biased by unsampled states, underestimating true entropy.
• Bayesian estimators address this by modeling the low-probability tail, but the driving factors remain unclear.

Purpose of the Study:

• To identify the statistical features of observed data that determine the tail model in Bayesian entropy estimators.
• To develop approximate analytical entropy estimators for undersampled distributions.
• To provide an intuitive understanding of how Bayesian entropy estimators function.

Main Methods:

• Analysis of well-known entropy estimators for discrete probability distributions.
• Derivation of approximate analytical estimators based on identified data statistics.
• Investigation of the influence of sample size, maximum likelihood estimate, and coincidence statistics.

Main Results:

• Key data statistics influencing tail modeling include sample size, maximum likelihood estimate, number of coincidences, and dispersion of coincidences.
• Approximate analytical entropy estimators were derived for undersampled distributions.
• The study clarifies the relationship between data statistics and Bayesian entropy estimation.

Conclusions:

• The structure of the low-probability tail in Bayesian entropy estimation is primarily governed by a few fundamental data statistics.
• The derived analytical estimators offer a practical approach for undersampled scenarios.
• This work enhances the interpretability of Bayesian entropy estimation in complex systems analysis.