Related Experiment Videos
Online learning from finite training sets and robustness to input bias
1University of Edinburgh, Department of Physics, Nicolson Street, EH8 9BE, Edinnurgh, UK. Peter_Sollich@ed.ac.uk
Neural Computation
|November 6, 1998
Summary
Online gradient descent learning with finite training data and non-infinitesimal learning rates is analyzed. Online learning shows robustness to input bias, outperforming offline methods, while performance is similar for unbiased inputs.
Area of Science:
- Machine Learning
- Computational Neuroscience
- Statistical Physics
Background:
- Gradient descent is a fundamental optimization algorithm in machine learning.
- Understanding the impact of finite training data and learning rates is crucial for practical applications.
- Online and offline learning paradigms have distinct characteristics and performance trade-offs.
Purpose of the Study:
- To analyze the generalization error of online gradient descent learning.
- To investigate the effects of finite training set size and learning rates on model performance.
- To compare the robustness and performance of online versus offline learning.
Main Methods:
- Exact analytical results for generalization error.
- Analysis of a linear neural network model with N weights.
- Training on p = alphaN examples to study finite-size effects.
Main Results:
- Derived time-dependent generalization error for online gradient descent.
- Demonstrated the influence of finite training set size (alpha) on optimal learning rate (eta).
- Showcased online learning's superior robustness to input bias compared to offline learning.
Conclusions:
- Finite training set size significantly impacts optimal learning rate selection.
- Online learning offers advantages in scenarios with biased input data.
- For unbiased data, online and offline learning exhibit comparable performance.