Direct Error-Driven Learning for Deep Neural Networks With Applications to Big Data.

Area of Science:

• Machine Learning
• Deep Neural Networks
• Big Data Analytics

Background:

• Heterogeneity and noise in big data increase generalization error in traditional deep neural networks (deep NNs).
• Vanishing gradients pose a significant challenge in training deep NNs, hindering effective learning.
• Existing learning regimes struggle to address both data noise and gradient issues simultaneously.

Purpose of the Study:

• To propose a direct error-driven learning (EDL) scheme to reduce generalization error in deep NNs.
• To overcome the challenge of vanishing gradients during the training of deep NNs.
• To effectively handle data heterogeneity and noise within the learning process.

Main Methods:

• Introduced a 'neighborhood' concept to reduce the impact of data heterogeneity and noise.
• Defined an overall error comprising learning and approximate generalization errors.
• Developed a novel NN weight-tuning law using a layer-wise performance measure and additional constraints for noisy dimensions.

Main Results:

• The proposed EDL scheme effectively mitigates the vanishing gradient problem.
• Demonstrated a 6% improvement in model generalization.
• Successfully addressed issues of data heterogeneity and noise in deep NN training.

Conclusions:

• The direct error-driven learning (EDL) scheme offers a robust solution for training deep neural networks (deep NNs) with noisy and heterogeneous big data.
• The method enhances model generalization and overcomes critical training challenges like vanishing gradients.
• This approach provides a significant advancement in deep learning for complex datasets.