Deep neural networks with a set of node-wise varying activation functions

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) are powerful tools for feature learning.
• However, understanding and utilizing the importance of learned features remains a challenge.
• Existing methods often require separate processes for feature selection and network optimization.

Purpose of the Study:

• To develop a novel deep neural network architecture capable of learning feature importance intrinsically.
• To enable automatic pruning of less important features, thereby reducing network complexity.
• To demonstrate that pruned networks can be retrained without significant performance degradation.

Main Methods:

• Introduction of deep neural networks with node-wise varying activation functions.
• Activation functions are designed to make nodes with smaller indices more sensitive during training.
• This sensitivity gradient results in features being sorted by node index according to their importance.

Main Results:

• The proposed networks successfully learn both input features and their relative importance.
• Nodes corresponding to less important features can be identified and pruned.
• Validation across shallow, deep, and transferred networks confirms the feature-sorting property and pruning effectiveness.

Conclusions:

• The novel DNN architecture effectively learns and ranks feature importance.
• Automatic feature pruning based on learned importance is feasible and efficient.
• This approach offers a pathway to more interpretable and computationally efficient deep learning models.