Weight Importance Is Not Enough: Neural Expressiveness for Model Compression

Neural Network Pruning has been established as driving force in the exploration of memory and energy efficient solutions with high throughput both during training and at test time. We propose Neural Expressiveness (NEXP), a new criterion for model compression that shifts the focus from weight importance to the capacity of neurons or groups of neurons to redistribute and preserve information flow, quantified through activation overlap. This perspective establishes a clear conceptual distinction between importance-based (weight-centric) and expressiveness-based (activation-centric) pruning, motivating a broader class of activation-dependent compression strategies. We provide a mathematical formulation of expressiveness that is model-agnostic and can be integrated into any activation-based pruning framework. Our analysis shows that expressiveness captures information complementary to weight importance, revealing a form of partial orthogonality between the two families of criteria. Furthermore, we demonstrate that expressiveness is strongly tied to the network's initialization, enabling a stateless pruning regime that is partially independent of training dynamics and directly informs the "when to prune" question. We also show that NEXP can be reliably approximated using limited arbitrary data, and thus paving the way for data-agnostic compression. Overall, the proposed concept of "Neural Expressiveness" in this paper expands on the prevalent yet unexplored notion of expressiveness in efficient deep learning and aims to forge a distinct paradigm, transitioning from the conventional weight-centered importance assessment to an emphasis on activations in model compression.