Refinements of Approximation Results of Conditional Restricted Boltzmann Machines

Conditional restricted Boltzmann machines (CRBMs) are the conditional variant of restricted Boltzmann machines (RBMs), which are used to simulate conditional probability distributions. While promising for practical applications, there is a lack of theoretical studies on the approximation ability of CRBMs. In this article, by contributing analysis tools, especially designed for the conditional models, we improve the results of the representational power of CRBMs based on existing work on RBMs. We first study the properties of CRBMs to approximate conditional Markov random fields. On this basis, a universal approximation result is obtained by deriving upper bounds on the minimal number of hidden units, which improves the previous result. Furthermore, the result about maximal approximation errors of CRBMs is also improved by reducing the number of hidden units that can guarantee approximations within a given error tolerance. Furthermore, the representational efficiency of CRBMs in computing deterministic conditional distributions is investigated. Specifically, we show that there are exponentially many deterministic conditional distributions that cannot be computed by CRBMs whose size does not exponentially grow with the number of input units. Some specific examples of these hard-to-present deterministic conditional distributions are provided. Finally, some numerical experiments are carried out to verify the theoretical results of the properties of CRBMs to approximate conditional Markov random fields and the universal approximation of CRBMs.