A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture.

Area of Science:

• Hardware implementation of neural network components
• Non-von Neumann computer architectures
• Integrated circuit design for AI acceleration

Background:

• Computing-In-Memory (CIM) offers energy efficiency and reduced latency for neural networks by moving computation closer to memory.
• Existing CIM research prioritizes multiply-and-accumulate (MAC) operations, neglecting high-precision hardware for non-linear activation functions like sigmoid.
• Current sigmoid implementations in hardware often use approximations, leading to significant errors and impacting neural network accuracy.

Purpose of the Study:

• To propose and validate a high-precision hardware circuit for the sigmoid activation function that exactly matches its mathematical expression.
• To address the limitations of approximate sigmoid implementations in terms of error, time overhead, and power consumption within CIM architectures.
• To demonstrate the effectiveness of the proposed high-precision sigmoid in a Computing-In-Memory based convolutional neural network.

Main Methods:

• Designed and simulated a novel high-precision sigmoid circuit using the SMIC 40 nm process.
• Evaluated the circuit's precision by comparing its output to the ideal sigmoid function, calculating maximum and average errors.
• Integrated the high-precision sigmoid circuit into a multi-layer convolutional neural network on a CIM architecture for handwritten digit recognition tasks.

Main Results:

• The proposed high-precision sigmoid circuit achieved maximum and average errors of 2.74% and 0.21% respectively, closely matching the ideal sigmoid function.
• A convolutional neural network employing the high-precision sigmoid in a CIM architecture demonstrated comparable handwritten digit recognition accuracy to software-based ideal sigmoid implementations.
• The CIM network achieved 97.06% accuracy with online training and 97.74% with offline training, validating the precision and effectiveness of the hardware sigmoid.

Conclusions:

• The developed high-precision sigmoid circuit provides an exact hardware implementation, overcoming the limitations of previous approximate methods.
• This advancement is crucial for enhancing the accuracy and efficiency of non-linear layers in Computing-In-Memory systems.
• The successful integration into a CIM-based neural network validates its potential for real-world applications in AI acceleration.