Parametric Deformable Exponential Linear Units for deep neural networks.

Area of Science:

• Computer Vision
• Deep Learning
• Neural Network Architectures

Background:

• Rectified activation units are crucial for deep neural network performance in computer vision.
• Existing activation functions have limitations in optimizing convergence speed and accuracy.

Purpose of the Study:

• To propose and theoretically validate a new activation function, the Parametric Deformable Exponential Linear Unit (PDELU).
• To enhance the convergence speed and accuracy of deep neural networks through improved activation response properties.

Main Methods:

• Theoretical verification of PDELU's effectiveness in promoting steepest descent during training.
• Empirical evaluation of PDELU within various Convolutional Neural Network (CNN) architectures.
• Comparative analysis against established and adaptive activation functions.

Main Results:

• PDELU demonstrated a higher convergence speed and improved accuracy across multiple datasets (CIFAR-10, CIFAR-100, ImageNet-2015).
• The proposed activation function outperformed existing methods, including shape-specific and shape-adaptive functions.
• PDELU integration enhanced performance in diverse CNN models like NIN, ResNet, WRN, and DenseNet.

Conclusions:

• The Parametric Deformable Exponential Linear Unit (PDELU) offers significant advantages for deep learning models.
• PDELU's flexible shape optimizes activation responses, leading to faster and more accurate training in computer vision applications.