Deep Neural Networks with Multistate Activation Functions.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep Neural Networks (DNNs) commonly use binary activation functions.
• Existing activation functions have limitations in representing complex data states.

Purpose of the Study:

• To introduce and evaluate Multistate Activation Functions (MSAFs) for DNNs.
• To explore the performance of MSAFs in classification and speech recognition tasks.

Main Methods:

• Development of N-order and symmetrical MSAFs.
• Training DNNs with MSAFs using conventional Stochastic Gradient Descent (SGD) and mean-normalised SGD.
• Evaluation on the TIMIT corpus for speech recognition.

Main Results:

• DNNs with MSAFs achieved a 5.60% relative improvement in phoneme error rates on speech recognition tasks.
• Mean-normalised SGD enhanced DNN training with MSAFs, particularly on large datasets.
• Direct training without pretraining yielded a 5.82% relative improvement in word error rates with large datasets.

Conclusions:

• MSAFs offer a significant advancement over conventional activation functions in DNNs.
• Mean-normalised SGD is an effective training method for DNNs employing MSAFs.
• MSAFs demonstrate strong potential for improving speech recognition and other classification tasks.