Explainable Structured Pruning of BERT via Mutual Information

Bidirectional Encoder Representations from Transformers (BERT) excels in natural language processing (NLP) but is costly on edge devices. We introduce an unsupervised, retraining-free structured pruning scheme for BERT, guided by mutual information (MI). Leveraging Rényi α-order entropy, we design a representation-aware MI estimator and a principled kernel-bandwidth selection, producing stable, sample-efficient neuron-level pruning signals. This method removes redundant units while preserving representational capacity, reduces memory and latency, and deploys readily on commodity hardware. Explainable-AI visualizations clarify how compression reshapes intermediate features and predictions. Across benchmarks, the compressed models maintain minimal accuracy loss, outperform or match strong unsupervised baselines, and remain competitive with supervised alternatives.