Scalable Empirical Bayes Inference and Bayesian Sensitivity Analysis

Consider a Bayesian setup in which we observe $$ , whose distribution depends on a parameter $$ , that is, $$ . The parameter $$ is unknown and treated as random, and a prior distribution chosen from some parametric family $$ , is to be placed on it. For the subjective Bayesian there is a single prior in the family which represents his or her beliefs about $$ , but determination of this prior is very often extremely difficult. In the empirical Bayes approach, the latent distribution on $$ is estimated from the data. This is usually done by choosing the value of the hyperparameter $$ that maximizes some criterion. Arguably the most common way of doing this is to let $$ be the marginal likelihood of $$ , that is, $$ , and choose the value of $$ that maximizes $$ . Unfortunately, except for a handful of textbook examples, analytic evaluation of $$ is not feasible. The purpose of this paper is two-fold. First, we review the literature on estimating it and find that the most commonly used procedures are either potentially highly inaccurate or don't scale well with the dimension of $$ , the dimension of $$ , or both. Second, we present a method for estimating $$ , based on Markov chain Monte Carlo, that applies very generally and scales well with dimension. Let $$ be a real-valued function of $$ , and let $$ be the posterior expectation of $$ when the prior is $$ . As a byproduct of our approach, we show how to obtain point estimates and globally-valid confidence bands for the family $$ , $$ . To illustrate the scope of our methodology we provide three detailed examples, having different characters.