On the relationship between deterministic and probabilistic directed Graphical models: from Bayesian networks to...

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Statistics

Background:

• Machine learning models like Bayesian Networks (BNs) and Recursive Neural Networks (RNNs) handle variable-size structured data (sequences, graphs).
• BNs use probabilistic relationships, while RNNs use deterministic, neural network-parameterized relationships for variables in directed acyclic graphs.

Purpose of the Study:

• To formally investigate the relationship between Bayesian Networks and Recursive Neural Networks.
• To demonstrate how RNNs can be understood as a limiting case of BNs under specific conditions.

Main Methods:

• Formal mathematical analysis of the relationship between BNs and RNNs.
• Investigating BNs with local conditional distributions converging to delta functions and vanishing covariance.
• Analyzing the behavior and exactness of Belief Propagation (BP) in deterministic BNs.

Main Results:

• Recursive Neural Networks (RNNs) are demonstrated to be distributional and probabilistic limits of Bayesian Networks (BNs) when source node variables are observed.
• Conditions for uniform convergence are established.
• The behavior and exactness of Belief Propagation (BP) in deterministic BNs are analyzed.

Conclusions:

• RNNs can be viewed as a deterministic limit of BNs, unifying these two classes of models.
• This provides implications for designing hybrid architectures combining probabilistic and deterministic elements.
• Insights into corresponding inference algorithms for these mixed architectures are discussed.