Explainable Structured Pruning of BERT via Mutual Information.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Bidirectional Encoder Representations from Transformers (BERT) models are powerful for natural language processing (NLP) but computationally expensive for edge devices.
• Existing compression methods often require extensive retraining or supervised data, limiting their applicability.

Purpose of the Study:

• To introduce an unsupervised, retraining-free structured pruning scheme for BERT models.
• To reduce the computational cost and memory footprint of BERT for deployment on edge devices.

Main Methods:

• A novel pruning scheme guided by mutual information (MI) using Rényi α-order entropy.
• Development of a representation-aware MI estimator and a principled kernel-bandwidth selection for stable, sample-efficient pruning signals.
• Application of Explainable-AI visualizations to understand feature and prediction changes post-compression.

Main Results:

• The proposed method effectively removes redundant units in BERT while preserving representational capacity.
• Compressed models show significant reductions in memory and latency, suitable for commodity hardware.
• Evaluations across benchmarks demonstrate minimal accuracy loss, outperforming unsupervised baselines and competing with supervised methods.

Conclusions:

• The unsupervised, retraining-free pruning scheme offers an efficient way to compress BERT models.
• This approach facilitates the deployment of advanced NLP models on resource-constrained edge devices.
• The method maintains model performance and provides insights into the compression process through Explainable-AI.