The Stochastic Delta Rule: Faster and More Accurate Deep Learning Through Adaptive Weight Noise

Area of Science:

• Deep learning
• Machine learning
• Artificial intelligence

Background:

• Multilayer neural networks excel in various tasks but face overfitting and misspecification issues.
• Dropout is a regularization technique that randomly removes hidden units to combat overfitting.
• Nonlinear parameter estimation in hierarchical models presents challenges like saddle points and colinearity.

Discussion:

• This research revisits the Stochastic Delta Rule (SDR), a 1990 parameter search method, demonstrating Dropout as a special case.
• SDR redefines weights as random variables, sampled during forward activation, creating numerous shared-weight networks.
• SDR incorporates prediction error into weight updates, injecting noise that reflects local history and averages models.

Key Insights:

• SDR acts as a local gradient-dependent simulated annealing, converging to a Bayes-optimal network.
• On ImageNet using DenseNet-BC 121, SDR reduced top-5 validation error by approximately 13% compared to Dropout.
• SDR achieved target accuracy in fewer epochs and showed up to 80% improvement in training error.

Outlook:

• SDR offers a more robust and efficient alternative to Dropout for deep neural networks.
• Further research can explore SDR's application in diverse deep learning architectures and complex datasets.
• SDR's adaptive, error-driven noise injection holds potential for advancing model generalization and convergence speed.