Analyzing the impact of KV representation compression on explainability in lightweight transformer-based sentiment...

Area of Science:

• Natural Language Processing
• Machine Learning Explainability

Background:

• Lightweight Transformer models like DistilBERT and MiniLM are crucial for efficient sentiment analysis.
• Understanding the explainability of these compressed models is essential for reliable deployment.

Purpose of the Study:

• To analyze the impact of key-value (KV) representation compression on the explainability of lightweight Transformer models.
• To evaluate how compression affects predictive performance and explanation fidelity across different datasets.

Main Methods:

• Utilized a two-stage compression framework to compress KV representations in DistilBERT and MiniLM.
• Evaluated models on sentiment analysis tasks using IMDB, SST-2, and TweetEval datasets.
• Assessed representational similarity, predictive performance (accuracy), and explanation preservation metrics (fidelity, stability, contrastive consistency).

Main Results:

• Achieved compression ratios of 2.46-3.05 with high representational similarity (>0.994).
• Maintained stable predictive performance with accuracy variations within 0.02.
• Explanation preservation varied by dataset: SST-2 showed high robustness (>0.90), while IMDB and TweetEval exhibited moderate reductions (0.53-0.74) but preserved core structures.

Conclusions:

• KV representation compression offers significant memory savings for lightweight Transformers without compromising predictive performance.
• Explainability is largely preserved, though the degree of preservation is dataset-dependent, influenced by factors like input length and structural complexity.