Cross Entropy in Deep Learning of Classifiers Is Unnecessary-ISBE Error Is All You Need

Area of Science:

• Deep Learning
• Machine Learning
• Computer Science

Background:

• Deep learning classifiers commonly use SoftMax and CrossEntropy cost functions.
• SoftMax converts model scores to class probabilities.
• CrossEntropy measures prediction divergence from target distributions.

Purpose of the Study:

• To introduce the Improved SoftMax-Before-Entropy (ISBE) functionality.
• To demonstrate the redundancy of cross-entropy computation in deep learning.
• To optimize deep learning classifier training and back-propagation.

Main Methods:

• Introduction of the ISBE functionality.
• Analysis of back-propagation without cross-entropy calculation.
• Implementation and testing on perceptron and convolutional networks using MNIST dataset.
• Formal proof of the SoftMax trick's applicability.

Main Results:

• ISBE omits cross-entropy calculation, simplifying back-propagation.
• Classifier performance is maintained using SoftMax only, with various activation functions.
• Significant operational savings observed in forward and backward passes.
• ISBE is effective with SoftMax, Sigmoid, Tanh, HardSigmoid, and HardTanh.

Conclusions:

• Cross-entropy computation is unnecessary for deep learning classifiers.
• ISBE offers computational savings and maintains performance.
• The proposed method is suitable for programmers and students designing deep models.