Bounds on the Excess Minimum Risk via Generalized Information Divergence Measures

Area of Science:

• Information Theory
• Machine Learning
• Statistical Inference

Background:

• Estimating a target vector Y from observed data X or its degraded version Z is crucial in various fields.
• The excess minimum risk quantifies the performance loss due to data degradation (Z vs. X).
• Existing bounds often rely on mutual information and specific distribution assumptions.

Purpose of the Study:

• To derive generalized upper bounds on the excess minimum risk for estimating Y from X or Z.
• To introduce novel bounds using Rényi and α-Jensen-Shannon divergences, generalizing prior work.
• To extend the applicability of these bounds to a wider range of joint distributions by relaxing constant sub-Gaussianity assumptions.

Main Methods:

• Utilizing generalized information divergence measures, including Rényi and α-Jensen-Shannon divergences.
• Analyzing a Markov chain Y→X→Z where Y is the target, X is the observed feature, and Z is the degraded version.
• Developing theoretical bounds that do not assume a constant sub-Gaussian parameter.

Main Results:

• A family of generalized upper bounds on the excess minimum risk was derived.
• The new bounds are shown to be applicable to broader classes of joint distributions compared to previous methods.
• Numerical examples demonstrate that the generalized divergence-based bounds can be tighter than mutual information-based bounds.

Conclusions:

• The developed generalized information divergence measures provide tighter and more broadly applicable bounds for excess minimum risk.
• These findings advance the understanding of information loss in data degradation and estimation tasks.
• The relaxed assumptions on sub-Gaussianity enhance the utility of these bounds in practical machine learning scenarios.