Automatic Learning Rate Adaption for Memristive Deep Learning Systems | JoVE Visualize

Area of Science:

Hardware acceleration for artificial intelligence
Emerging electronic device applications
Deep learning system optimization

Background:

Memristors offer a promising hardware solution for efficient and compact deep learning (DL) systems, enhancing hybrid complementary metal-oxide-semiconductor (CMOS) technology.
Traditional DL systems often require manual tuning of learning rates, which can be time-consuming and suboptimal.
Variations in memristive devices (cycle-to-cycle and device-to-device) pose challenges for reliable DL system implementation.

Purpose of the Study:

To present an automatic learning rate tuning method for memristive DL systems.
To enable adaptive learning rates in deep neural networks (DNNs) using memristive devices.
To address challenges like noisy gradients, variations, and overfitting in memristive DL systems for image recognition.

Main Methods:

Memristive devices are employed to dynamically adjust the learning rate within DNNs during the backpropagation (BP) algorithm.
The memristance/conductance adjustment process inherently creates a fast-then-slow adaptation speed for the learning rate.
Fuzzy control methods are integrated for adaptive learning, specifically targeting pattern recognition and mitigating overfitting.

Main Results:

The proposed method eliminates the need for manual learning rate tuning in the adaptive BP algorithm.
The system demonstrates robustness against noisy gradients, diverse network architectures, and varied datasets, despite inherent memristor variations.
The integration of a quantized neural network architecture significantly boosts training efficiency without compromising testing accuracy.

Conclusions:

This work presents the first memristive DL system utilizing an adaptive learning rate for image recognition.
The developed system offers a robust and efficient solution for memristive DL, overcoming common challenges.
The combination of adaptive learning rates and quantized networks in memristive DL systems paves the way for highly efficient AI hardware.