Enabling Training of Neural Networks on Noisy Hardware

Area of Science:

• Artificial Intelligence
• Machine Learning Hardware
• Deep Neural Networks

Background:

• Conventional training algorithms like Stochastic Gradient Descent (SGD) struggle with non-ideal analog hardware due to symmetry requirements.
• Existing Tiki-Taka algorithm addresses symmetry but imposes stringent hardware demands.
• Analog hardware offers potential power and speed benefits for deep neural network (DNN) training.

Purpose of the Study:

• To introduce a more robust algorithm, TTv2, that further relaxes hardware requirements for DNN training on analog devices.
• To enhance noise tolerance in analog hardware-based DNN training.
• To enable efficient model extraction and deployment from analog hardware.

Main Methods:

• Developed the TTv2 algorithm, an advancement of the Tiki-Taka approach.
• Incorporated lightweight digital filtering operations outside analog arrays to complement the core algorithm.
• Implemented a model extraction technique for transferring trained neural networks to other hardware platforms.

Main Results:

• TTv2 significantly reduces the required number of device conductance states (from thousands to tens).
• Achieved approximately 100x increased noise tolerance for device conductance modulations and 10x for matrix-vector multiplication.
• Empirical simulations demonstrate TTv2's ability to train DNNs to near-ideal accuracy even in highly noisy hardware conditions.

Conclusions:

• TTv2 provides an end-to-end training and model extraction solution for extremely noisy crossbar-based analog hardware.
• The algorithm minimizes implementation costs while retaining the power and speed advantages of analog hardware.
• Extracted models show improved test accuracy, surpassing the performance of the originally trained model.