An Adaptive Deep Belief Network With Sparse Restricted Boltzmann Machines

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep Belief Networks (DBNs) are effective for nonlinear data but suffer from dense representation and local optima during backpropagation.
• Designing robust DBNs is challenging due to traditional limitations in representation and training.

Purpose of the Study:

• To introduce a novel Deep Belief Network (DBN) model, ARP-DBN, that overcomes limitations of traditional DBNs.
• To improve model robustness and accuracy in learning unknown data representations, particularly for nonlinear systems.

Main Methods:

• Developed an Adaptive Sparse Restricted Boltzmann Machine (AS-RBM) with adaptive learning step size and regularization for sparse representation.
• Employed Partial Least Square (PLS) regression for layer-by-layer fine-tuning of initial weights, optimizing from output to input layers.
• Provided convergence and stability analysis for the proposed ARP-DBN method.

Main Results:

• The ARP-DBN model demonstrated superior learning accuracy and faster learning speeds compared to existing methods.
• Achieved robust model performance in Mackey-Glass time-series prediction, 2-D function approximation, and unknown system identification.
• Validated the effectiveness of AS-RBM and PLS fine-tuning in enhancing DBN capabilities.

Conclusions:

• The proposed ARP-DBN model offers a more robust and accurate approach to deep belief network construction.
• The integration of AS-RBM and PLS regression effectively addresses challenges associated with DBN training and representation.
• ARP-DBN shows significant potential for applications requiring accurate modeling of complex, nonlinear systems.