Analyzing the impact of KV representation compression on explainability in lightweight transformer-based sentiment...

This study analyzes the impact of key-value (KV) representation compression on explainability in lightweight Transformer-based sentiment analysis. DistilBERT and MiniLM are evaluated on IMDB, SST-2, and TweetEval using a two-stage compression framework. The proposed approach achieves compression ratios of 2.46-3.05 while maintaining high representational similarity (cosine similarity > 0.994). Predictive performance remains stable across all settings, with accuracy variations within 0.02. From an explainability perspective, compression effects vary depending on dataset characteristics. SST-2 demonstrates strong robustness, maintaining explanation preservation above 0.90 with high contrastive consistency (~ 0.89). In contrast, IMDB and TweetEval show moderate reductions in fidelity and stability, although meaningful explanation structures are still preserved (preservation ~ 0.53-0.74). In some cases, fidelity and matching reliability are slightly improved after compression. These results indicate that KV representation compression improves memory efficiency while largely preserving predictive performance and core explanation structures. However, its impact on explainability depends on dataset characteristics such as input length and structural complexity.