A fast and scalable recurrent neural network based on stochastic meta descent

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Recurrent Neural Networks (RNNs) are powerful tools for sequential data processing.
• Traditional online learning algorithms for RNNs, like Real-Time Recurrent Learning (RTRL), often suffer from high computational and storage complexity.
• Scalability remains a significant challenge for applying RNNs to complex, large-scale problems.

Purpose of the Study:

• To develop an efficient and scalable online learning algorithm for RNNs.
• To reduce the computational and storage complexity of existing RTRL algorithms.
• To accelerate the learning process in RNNs while maintaining or improving performance.

Main Methods:

• The proposed algorithm is based on Real-Time Recurrent Learning (RTRL) but reduces the sensitivity set of each neuron.
• It achieves a reduced storage and computational complexity of O(N(2)).
• Stochastic Meta Descent (SMD) is employed for adaptive step-size control, incorporating curvature information to accelerate learning. A clustered version is introduced for enhanced scalability.

Main Results:

• The novel algorithm demonstrates a dramatic reduction in resource requirements compared to standard RTRL.
• Simulation results show the approach outperforms regular RTRL by nearly an order of magnitude.
• The localized learning framework is amenable to parallel hardware realization.

Conclusions:

• The developed online learning algorithm offers significant efficiency and scalability improvements for RNNs.
• The method accelerates learning and reduces resource demands without compromising performance.
• Its inherent parallelizability makes it suitable for hardware implementation, paving the way for more advanced RNN applications.