A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Neuroscience

Background:

• Neural network training involves various strategies, each with trade-offs in storage, determinism, and computational efficiency.
• Incremental learning methods like online, cyclic, and almost-cyclic learning require less storage than batch-mode learning but introduce stochasticity.

Purpose of the Study:

• To compare the performance of different neural network learning strategies: batch-mode, online, cyclic, and almost-cyclic learning.
• To analyze the deviations from ideal behavior in stochastic learning processes and quantify their asymptotic misadjustment.
• To determine the optimal learning steps required for achieving desired weight accuracy under different learning parameters.

Main Methods:

• Derivation of differential equations using stochastic methods to describe deviations from ideal behavior for small learning parameters.
• Computation of asymptotic misadjustment as a function of learning parameter and training patterns.
• Calculation of learning steps needed to reach a specific weight accuracy.

Main Results:

• All learning strategies approximate an 'ideal behavior' in the zeroth order (eta -> 0).
• Stochastic methods reveal differential equations for lowest-order deviations from ideal behavior.
• Asymptotic misadjustment was computed, and typical learning steps were calculated for fixed and time-dependent learning parameters.

Conclusions:

• Almost-cyclic learning (random cycles) is a more effective alternative to batch-mode learning compared to cyclic learning (fixed cycles).
• The study provides a quantitative framework for understanding and optimizing neural network training strategies.