Related Experiment Videos
Subset-based training and pruning of sigmoid neural networks
1Department of Electrical Engineering, Arizona State University, Tempe, USA
Summary
New algorithms for sigmoid network training, subset-based training (SBT) and subset-based training and pruning (SBTP), offer efficient memory use and combined training/pruning. These methods show comparable convergence to standard techniques.
Area of Science:
- Computational neuroscience
- Machine learning algorithms
- Artificial neural networks
Background:
- Sigmoid network training often involves rank-deficient Jacobian matrices.
- Standard training methods like Levenberg-Marquardt (LM) can be memory-intensive.
Purpose of the Study:
- To develop novel algorithms for efficient sigmoid network training.
- To reduce memory requirements compared to existing methods.
- To integrate network training and pruning into a single process.
Main Methods:
- Developed subset-based training (SBT) and subset-based training and pruning (SBTP) algorithms.
- Utilized the rank deficiency of Jacobian matrices in sigmoid networks.
- Implemented trust-region methods for both algorithms.
- Divided weight vectors based on Jacobian rank sizes during training.
Main Results:
- SBT and SBTP demonstrate similar convergence properties to the Levenberg-Marquardt (LM) method.
- Both algorithms achieve reduced memory requirements compared to LM.
- SBTP effectively combines network training and pruning into one procedure.
- Evaluated effectiveness through three examples and comparisons with existing algorithms.
Conclusions:
- SBT and SBTP offer efficient alternatives for sigmoid network training.
- The proposed methods provide comparable convergence with improved memory efficiency.
- SBTP presents a unified approach for training and pruning neural networks.