Incremental Online Learning of Randomized Neural Network With Forward Regularization.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Online learning for deep neural networks faces issues like delayed updates, high costs, and catastrophic forgetting.
• Existing methods often require retrospective retraining, hindering real-time decision-making.

Purpose of the Study:

• To propose a novel Incremental Online Learning (IOL) framework for Randomized Neural Networks (Randomized NN).
• To enable progressive, immediate decision-making and continuous performance improvement in online scenarios.

Main Methods:

• Developed an IOL framework for Randomized NN, including IOL with ridge regularization (-R) and IOL with forward regularization (-F).
• Derived incremental algorithms for -R/-F on non-stationary batch streams with recursive weight updates and variable learning rates.
• Theoretically derived relative cumulative regret bounds for -R/-F learners under adversarial assumptions.

Main Results:

• Both -R and -F frameworks avoid retrospective retraining and catastrophic forgetting.
• -F demonstrated improved learning performance by utilizing future unlabeled data and reducing online regrets compared to -R.
• Theoretical analysis and empirical validation showed superior online learning acceleration and reduced regret bounds with -F.

Conclusions:

• The proposed IOL frameworks for Randomized NN are effective for continuous learning and analytics.
• Forward regularization (-F) offers significant advantages over ridge regularization (-R) in online learning scenarios, particularly for long-term time-series forecasting and continual learning.