Computation of Kullback-Leibler Divergence in Bayesian Networks

Area of Science:

• Computational Statistics
• Machine Learning
• Probabilistic Graphical Models

Background:

• Kullback-Leibler (KL) divergence is a standard error measure for comparing probability distributions.
• Efficient computation is vital for tasks like approximate computation and probability learning.
• Direct KL divergence computation is often infeasible for high-dimensional Bayesian networks.

Purpose of the Study:

• To develop an efficient algorithm for computing KL divergence between probability distributions from distinct Bayesian networks.
• To address the computational challenges posed by high-dimensional probability distributions in Bayesian networks.
• To provide a practical implementation using Python and pgmpy.

Main Methods:

• Utilized an auxiliary deletion algorithm to compute marginal distributions.
• Implemented a cache for operations with potentials to reuse computations.
• Tested algorithms on Bayesian networks from the bnlearn repository.

Main Results:

• Demonstrated efficient computation of KL divergence for Bayesian networks with different structures.
• The caching mechanism significantly reduces redundant calculations.
• The developed Python code, based on pgmpy, offers a practical solution.

Conclusions:

• The proposed method enables efficient KL divergence computation between complex Bayesian networks.
• This advancement is beneficial for probabilistic inference and model comparison in high dimensions.
• The open-source implementation facilitates broader adoption and further research.