Training LSTM Networks With Resistive Cross-Point Devices

Area of Science:

• Neuromorphic engineering
• Artificial intelligence hardware
• Deep learning acceleration

Background:

• Resistive processing unit (RPU) devices offer power and speed advantages for training deep fully connected and convolutional neural networks.
• Recurrent neural networks (RNNs), particularly LSTMs, present unique challenges for hardware acceleration due to their sequential nature.

Purpose of the Study:

• To extend the RPU concept for efficient training of recurrent neural networks (RNNs).
• To investigate the impact of device imperfections and system parameters on RNN training using RPUs.
• To evaluate the feasibility of RPU-based training for large-scale RNN models.

Main Methods:

• Mapping recurrent layers to RPU device arrays.
• Simulating RPU-based training of LSTMs with varying device imperfections (e.g., symmetry, input resolution) and system parameters.
• Comparing RPU training performance against ideal floating-point training.

Main Results:

• RPU concept is applicable to RNNs, offering potential acceleration similar to other network types.
• Update symmetry is critical for RNNs; even minor asymmetry increases test error.
• Minimum 7-bit input resolution is required, reducible to 5 bits using stochastic rounding.
• RPU device variations and noise mitigate overfitting, reducing reliance on dropout.
• Simulations included large-scale networks, significantly larger than typical MLP models.

Conclusions:

• RPUs show promise for accelerating RNN training, including LSTMs.
• Careful management of device parameters like symmetry and input resolution is necessary for optimal performance.
• RPU hardware characteristics can inherently provide regularization benefits, simplifying training pipelines.