Asymptotic Distribution of Certain Types of Entropy under the Multinomial Law

Area of Science:

• Information Theory
• Statistical Inference
• Quantitative Social Science

Background:

• Rényi and Tsallis entropies are generalizations of Shannon entropy, crucial for quantifying uncertainty in complex systems.
• Fisher information measures the amount of information a random variable carries about an unknown parameter.
• Maximum likelihood estimation (MLE) is a standard method for estimating probability distributions from data.

Purpose of the Study:

• To derive asymptotic distributions for Rényi and Tsallis entropies and Fisher information using MLE from multinomial samples.
• To develop and validate statistical tests for comparing entropies across different samples, even with varying categories.
• To apply these novel statistical methods to real-world social survey data.

Main Methods:

• Derivation of asymptotic distribution expressions for selected information measures.
• Simulation studies to verify the accuracy of the derived asymptotic models (including normal approximations for Tsallis and Fisher).
• Development of test statistics for comparing entropies from two samples.
• Application of the developed tests to social survey data.

Main Results:

• Asymptotic distributions for Rényi and Tsallis entropies and Fisher information were successfully obtained.
• The derived asymptotic models, particularly the normal approximations for Tsallis entropy and Fisher information, demonstrated good fit with simulated data.
• Novel test statistics were developed, enabling comparisons of different entropy types across samples with unequal numbers of categories.
• Application to social survey data yielded results consistent with, yet more general than, traditional chi-squared tests.

Conclusions:

• The derived asymptotic models provide a robust framework for analyzing Rényi and Tsallis entropies and Fisher information.
• The developed statistical tests offer a more flexible and general approach for entropy comparison in empirical studies.
• These methods enhance the analysis of complex data, such as that found in social surveys.