Rényi Cross-Entropy Measures for Common Distributions and Processes with Memory.

Area of Science:

• Information Theory
• Machine Learning
• Statistical Inference

Background:

• Shannon cross-entropy is a fundamental concept in information theory.
• Rényi-type cross-entropies offer generalizations with applications in deep learning.
• Generative Adversarial Networks (GANs) benefit from improved loss functions.

Purpose of the Study:

• To derive closed-form expressions for Rényi and Natural Rényi differential cross-entropy.
• To analyze these measures for common continuous distributions within the exponential family.
• To summarize cross-entropy rates for Gaussian processes and Markov sources.

Main Methods:

• Derivation of analytical expressions for differential cross-entropy measures.
• Tabulation of results for exponential family distributions.
• Summarization of cross-entropy rates for specific stochastic processes.

Main Results:

• Closed-form solutions for Rényi and Natural Rényi differential cross-entropy are presented.
• Results are provided for a broad class of exponential family distributions.
• Cross-entropy rates for stationary Gaussian processes and finite-alphabet Markov sources are summarized.

Conclusions:

• The derived measures provide valuable tools for deep learning applications, particularly GANs.
• Tabulated results facilitate practical application and comparison of different distributions.
• The study enhances understanding of information-theoretic rates in complex systems.