Learning Optimized Structure of Neural Networks by Hidden Node Pruning With L₁ Regularization.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Neuroscience

Background:

• Multilayer perceptron (MLP) networks are fundamental in machine learning.
• Determining the optimal number of hidden nodes is crucial for network performance and preventing overfitting.
• L1 regularization is a technique used to encourage sparsity and simplify models.

Purpose of the Study:

• To propose and evaluate three distinct methods for determining the optimal number of hidden nodes in MLPs using L1 regularization.
• To introduce a matrix-based convergence proving method (MCPM) for analyzing the proposed algorithms.
• To assess the pruning efficiency and generalization capabilities of the developed methods.

Main Methods:

• Three L1 regularization-based methods are proposed: two using multiplier functions/multipliers on hidden-layer nodes, and one employing a smoothing approximation.
• A matrix-based convergence proving method (MCPM) is introduced to prove weak and strong convergence of smoothing algorithms.
• The methods involve training networks and discarding redundant hidden nodes to achieve optimal architecture.

Main Results:

• The proposed pruning methods demonstrated efficient pruning abilities across 11 diverse classification datasets.
• The methods achieved competitive generalization performance compared to existing approaches.
• Theoretical results concerning convergence were validated by empirical findings.

Conclusions:

• The developed L1 regularization techniques effectively optimize MLP architectures by pruning unnecessary hidden nodes.
• The proposed methods offer efficient solutions for network design, leading to improved generalization.
• The MCPM provides a robust theoretical foundation for the convergence analysis of these algorithms.