Pathfinder: Parallel quasi-Newton variational inference

Area of Science:

• Computational Statistics
• Bayesian Inference
• Machine Learning

Background:

• Variational inference methods are crucial for approximating complex probability distributions in machine learning and statistics.
• Existing methods like Automatic Differentiation Variational Inference (ADVI) and Hamiltonian Monte Carlo (HMC) have limitations in terms of accuracy and computational cost.
• Efficiently sampling from differentiable probability densities is a key challenge in many scientific domains.

Purpose of the Study:

• To introduce Pathfinder, a novel variational method for approximate sampling from differentiable probability densities.
• To demonstrate Pathfinder's effectiveness in approximating target distributions and generating accurate samples.
• To highlight Pathfinder's computational advantages over existing sampling techniques.

Main Methods:

• Pathfinder employs a quasi-Newton optimization path to find normal approximations to the target density.
• Local covariance is estimated using inverse Hessian information from the optimizer.
• The method selects the approximation with the lowest estimated Kullback-Leibler (KL) divergence to the target distribution.

Main Results:

• Pathfinder's approximate draws were superior to ADVI and comparable to short dynamic HMC chains, as measured by 1-Wasserstein distance.
• Pathfinder required one to two orders of magnitude fewer log density and gradient evaluations than ADVI and HMC.
• Importance resampling with Pathfinder further improved sample diversity and robustness, reducing 1-Wasserstein distance.

Conclusions:

• Pathfinder provides a computationally efficient and accurate alternative for approximate sampling from probability densities.
• The method's parallelizability offers significant speed advantages, especially on multi-core systems.
• Pathfinder shows promise for applications requiring fast and reliable approximate Bayesian inference.