A new pruning heuristic based on variance analysis of sensitivity information

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Neural network (NN) architecture selection is crucial for balancing performance and computational cost.
• Sensitivity analysis is an established technique for pruning parameters in feedforward NNs.

Purpose of the Study:

• To present a new pruning algorithm for neural networks.
• To quantify the relevance of input and hidden units using sensitivity analysis.

Main Methods:

• A novel pruning algorithm is proposed, leveraging sensitivity analysis.
• A statistical pruning heuristic based on variance analysis is introduced to identify and remove irrelevant units.
• The core idea is to prune parameters with sensitivity variance not significantly different from zero.

Main Results:

• Experimental results demonstrate the algorithm's effectiveness in correctly pruning irrelevant input and hidden units.
• The new pruning algorithm's performance is benchmarked against standard pruning techniques.

Conclusions:

• The developed pruning algorithm successfully identifies and removes redundant units in neural networks.
• This approach offers an effective method for optimizing neural network architectures.