Far from Asymptopia: Unbiased High-Dimensional Inference Cannot Assume Unlimited Data

Area of Science:

• Statistical modeling
• Information geometry
• Bayesian inference

Background:

• Inference from limited data necessitates a measure on parameter space, often a prior distribution in Bayesian analysis.
• Jeffreys prior, a common uninformative choice derived from information geometry, is shown to introduce significant bias in high-dimensional models.

Purpose of the Study:

• To address the bias introduced by Jeffreys prior in high-dimensional models.
• To propose a principled measure that yields unbiased posteriors by focusing on relevant parameters.

Main Methods:

• Demonstrating the bias of Jeffreys prior in models with effective dimensionality lower than the number of microscopic parameters.
• Developing and presenting a new optimal prior based on relevant parameters and the quantity of data.

Main Results:

• Jeffreys prior exhibits enormous bias in typical high-dimensional scientific models due to unequal treatment of parameters.
• The proposed optimal prior avoids this bias by focusing on relevant parameters, leading to unbiased posteriors.
• This optimal prior is data-dependent and converges to Jeffreys prior only in the asymptotic limit.

Conclusions:

• Standard uninformative priors like Jeffreys prior are inadequate for high-dimensional models common in science.
• A novel, data-dependent prior focusing on relevant parameters offers a principled solution for unbiased Bayesian inference.
• Achieving the asymptotic limit for Jeffreys prior requires an infeasible amount of data in many practical scenarios.