Incremental Online Learning of Randomized Neural Network With Forward Regularization

Online learning of deep neural networks faces challenges such as delayed non-incremental updating, increasing consumption, retrospective retraining, and catastrophic forgetting. To alleviate these drawbacks and achieve progressive immediate decision-making, we propose a novel Incremental Online Learning (IOL) framework of Randomized Neural Networks (Randomized NN), facilitating continuous improvements and analytics to Randomized NN performance in online scenarios. Within the framework, we further formulate IOL with ridge regularization (-R) and IOL with forward regularization (-F), both avoiding retrospective retraining and catastrophic forgetting. Moreover, the incremental algorithms for -R/-F on non-stationary batch stream are derived, featuring recursive weight updates and variable learning rates. Compared to -R, we recommend -F which improves learning performance using future unlabeled observations while further reducing online regrets to offline global experts. Additionally, we conduct a detailed analysis and theoretically derive relative cumulative regret bounds of the Randomized NN learners for -R/-F under adversarial assumptions via a novel methodology and present several corollaries, from which we observed the superiority in online learning acceleration and declined regret bounds of employing -F in IOL. Finally, our proposed methods were rigorously examined across diverse tasks, from simulation, regression, and classification tasks, to long-term time-series forecasting (LTSF) and continual learning (CL) fields, which distinctly validated the efficacy of the IOL frameworks and the advantages of forward regularization.