Error-aware probabilistic training for memristive neural networks.

Area of Science:

• Neuromorphic Engineering
• Materials Science
• Computer Science

Background:

• Analog computing-in-memory devices offer high energy efficiency by utilizing physical laws for computation.
• Stochastic device characteristics in analog hardware conflict with deterministic training algorithms like backpropagation (BP), hindering performance.
• A mismatch exists between analog device physics and traditional deep learning training methods.

Purpose of the Study:

• To propose and validate an error-aware probabilistic update (EaPU) method for training analog computing-in-memory devices.
• To address the algorithm-device mismatch in analog hardware training.
• To demonstrate significant energy reduction and performance improvement in deep neural network training on memristor systems.

Main Methods:

• Developed an error-aware probabilistic update (EaPU) method utilizing device writing noise for weight updates.
• Experimentally validated EaPU on a 180nm memristor system for image denoising and super-resolution tasks.
• Simulated EaPU performance on deep learning models including ResNet and Vision Transformers.

Main Results:

• EaPU reduced weight updates to less than 1‰ of traditional BP with minimal accuracy loss.
• EaPU training achieved over 60% accuracy improvement on memristor systems.
• EaPU demonstrated substantial energy savings: ~50.54× and 13.23× lower training energy compared to BP-based memristor training and MADEM, respectively.
• EaPU-based memristor hardware achieved nearly 6 orders of magnitude lower training energy than GPUs.

Conclusions:

• EaPU provides a precise and efficient method for training analog device-based deep neural networks.
• The proposed method effectively bridges the gap between stochastic analog hardware and deterministic training algorithms.
• EaPU represents a significant advancement for energy-efficient neuromorphic computing and AI hardware.