Understanding and mitigating noise in trained deep neural networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Hardware Engineering

Background:

• Deep neural networks (DNNs) have advanced AI capabilities, driven by specialized hardware like GPUs and TPUs.
• Current hardware emulates neural networks using binary computing, leading to high energy consumption and limited speed.
• Analog hardware offers potential for parallel and faster computation but faces challenges with neuron noise and its accumulation.

Purpose of the Study:

• To analyze noise propagation in deep neural networks with noisy nonlinear neurons in fully connected layers.
• To develop analytical methods for predicting noise levels in trained deep neural networks.
• To identify design criteria for noise-resilient analog neural network hardware.

Main Methods:

• Investigated additive, multiplicative, correlated, and uncorrelated noise in DNNs.
• Developed analytical models to predict noise propagation in symmetric and backpropagation-trained DNNs.
• Analyzed the impact of neuron activation function slopes on noise accumulation.

Main Results:

• Noise accumulation in deep neural networks is generally bounded and does not indefinitely degrade the signal-to-noise ratio with increased layers.
• Noise accumulation can be effectively suppressed by using neuron activation functions with a slope smaller than unity.
• Established a framework for understanding noise in analog fully connected deep neural networks.

Conclusions:

• Analog neural network hardware is feasible, with noise accumulation being a predictable and manageable challenge.
• Designing neural network hardware with specific activation function properties can mitigate noise issues.
• This research provides criteria for engineering noise-resilient analog neural network systems.