Related Experiment Videos
Deep, narrow sigmoid belief networks are universal approximators
Ilya Sutskever1, Geoffrey E Hinton
1Department of Computer Science, University of Toronto, Toronto, Ontario M55 3G4, Canada. ilya@cs.utoronto.ca
Neural Computation
|June 7, 2008
Summary
Exponentially deep belief networks can approximate any binary vector distribution with high accuracy. These networks are learnable using a greedy approach, though the method is currently impractical.
Area of Science:
- Machine Learning
- Artificial Intelligence
- Deep Learning
Background:
- Deep belief networks (DBNs) are probabilistic graphical models used for unsupervised learning.
- Understanding the representational capacity of DBNs is crucial for advancing deep learning theory.
- Previous research has explored the expressivity of various neural network architectures.
Purpose of the Study:
- To investigate the approximation capabilities of exponentially deep belief networks.
- To determine if DBNs with limited layer width can accurately represent complex data distributions.
- To explore the feasibility of learning such networks through a greedy algorithm.
Main Methods:
- Theoretical analysis of exponentially deep belief networks.
- Mathematical proofs demonstrating approximation accuracy.
- Development of a greedy learning strategy for DBNs.
Main Results:
- Exponentially deep belief networks can approximate any binary vector distribution to arbitrary accuracy.
- This approximation holds even when the width of each network layer is constrained by the data dimensionality.
- A greedy learning method for these networks was identified, though it is computationally demanding.
Conclusions:
- Deep belief networks possess significant representational power, capable of modeling complex data distributions.
- The architecture allows for high-fidelity approximation even with restricted layer widths.
- While a greedy learning approach exists, practical implementation challenges remain.
Related Concept Videos
01:23Linear Approximations
For a differentiable function of two variables, linear approximation estimates values near a known point by replacing the curved surface with its tangent plane. Consider the function\begin{equation*}f(x,y)=x^2+3y^2\end{equation*}near the point (2, 1). The exact value at this point is f(2, 1) = 22 + 3(1)2 = 4 + 3 = 7.The linear approximation of f(x, y)) near (a, b) is\begin{equation*}L(x,y)=f(a,b)+f_x(a,b)(x-a)+f_y(a,b)(y-b)\end{equation*}First, compute the partial derivatives: fx(x, y) = 2x and...
01:26Linearization and Approximation
Linearization is a mathematical technique used to approximate complex, nonlinear functions with simpler linear models in the vicinity of a chosen reference point. The method is based on the idea that, although a function may be difficult to evaluate exactly, its behavior near a specific input value can often be closely approximated by the tangent line at that point. This approach is particularly useful when small deviations from a known value are involved.Consider the square root function, for...