CRESPR: Modular sparsification of DNNs to improve pruning performance and model interpretability.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Modern deep neural networks (DNNs) possess numerous parameters, leading to high computational and memory costs.
• Model pruning offers a solution to reduce complexity, decrease resource demands, and improve model interpretability.
• Existing pruning methods may struggle to maintain performance at extreme sparsity levels.

Purpose of the Study:

• To introduce a novel pruning algorithm, Cluster-Restricted Extreme Sparsity Pruning of Redundancy (CRESPR).
• To enhance pruning efficiency and achieve better performance, especially at high pruning ratios.
• To improve the interpretability of deep neural network models.

Main Methods:

• Developed CRESPR, a pruning algorithm that partitions neural networks into modular units.
• Utilized the Hessian matrix to provide an analytic explanation for the performance benefits of modular structures in sparse DNNs.
• Designed modular units with predominantly internal connections to reveal feature processing pathways.

Main Results:

• CRESPR demonstrates superior pruning performance compared to state-of-the-art methods at high and extremely high pruning ratios.
• The modular structure of CRESPR facilitates a clearer understanding of how feature subgroups are processed within the DNN.
• Experimental validation across multiple DNN architectures and datasets confirmed CRESPR's effectiveness.

Conclusions:

• CRESPR offers an effective strategy for pruning deep neural networks, achieving high efficiency and performance.
• The algorithm significantly enhances model interpretability by exposing internal processing mechanisms.
• CRESPR represents a promising advancement for developing more efficient and understandable AI models.