Fast Approximations of Activation Functions in Deep Neural Networks when using Posit Arithmetic

Area of Science:

• Computer Science
• Numerical Analysis
• Artificial Intelligence

Background:

• Deep Neural Networks (DNNs) face real-time processing constraints.
• The IEEE 754 floating-point standard presents limitations for efficient hardware implementation.
• The Posit number system offers a promising alternative for faster, hardware-friendly information representation.

Purpose of the Study:

• To explore the potential of the Posit number system for accelerating DNNs.
• To develop novel arithmetic operators for Posits suitable for current hardware (ALUs).
• To evaluate the performance and accuracy of Posit-based DNNs compared to traditional floating-point representations.

Main Methods:

• Introduced a new class of Posit operators, termed L1 operators, using integer arithmetic for approximated operations.
• Focused on Posit configurations with zero exponent bits.
• Emulated Posit arithmetic using existing Arithmetic-Logic Units (ALUs) and their vectorized operations.
• Tested Posit-based activation functions in DNNs, comparing various bit-width Posits against 32-bit floats.

Main Results:

• L1 operators provide faster computation than exact counterparts with negligible accuracy degradation.
• Posit operations can be efficiently emulated on ALUs, enabling vectorization.
• 10-bit to 16-bit Posits serve as effective replacements for 32-bit floats in DNNs.
• 8-bit Posits present a compelling trade-off between performance, storage, and accuracy for DNNs.

Conclusions:

• The Posit number system, particularly with L1 operators, can significantly speed up DNNs on current hardware.
• Posits offer a viable, efficient alternative to floating-point numbers for AI applications, especially with limited hardware support.
• Smaller Posit formats (up to 14 bits) are highly advantageous for future widespread adoption and efficient tabulation of operations.