Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices.

Area of Science:

• Computer Science
• Electrical Engineering
• Materials Science

Background:

• Memristor devices offer high capacity for machine learning accelerators but suffer from non-idealities hindering software-equivalent online training accuracy.
• Existing methods struggle with accuracy due to device limitations during gradient-based training.

Purpose of the Study:

• To improve the accuracy of memristor-based online training for machine learning accelerators.
• To reduce memory overhead during mini-batch training in memristor systems.
• To investigate efficient gradient matrix reconstruction methods for memristor arrays.

Main Methods:

• Employed Mini-Batch Gradient Descent (MBGD) with stochastic rounding to mitigate vanishing weight updates.
• Utilized decomposition methods, specifically streaming batch Principal Component Analysis (PCA) and Non-negative Matrix Factorization (NMF), for gradient matrix reconstruction.
• Evaluated internal (rank-seq) and external (rank-sum) reconstruction strategies for memristor arrays.
• Trained a memristor-based multi-layer perceptron on the MNIST database.

Main Results:

• Streaming batch PCA and NMF achieved near MBGD accuracy in memristor-based perceptrons.
• Effective accuracy was maintained with low rank (3-10) decomposition, yielding substantial memory savings.
• NMF with rank-seq outperformed streaming batch PCA with rank-seq at low ranks.

Conclusions:

• Matrix decomposition methods, particularly NMF rank-seq, can enable high-accuracy online training in memristor accelerators with reduced memory footprint.
• NMF rank-seq demonstrates suitability for hardware implementation in future memristor-based accelerators, overcoming non-idealities.
• The proposed methods bridge the gap between memristor potential and practical, high-accuracy machine learning applications.