Distilling Transformers: A Survey

Transformers have gained widespread recognition across sectors and academic fields for their innovative self-attention mechanism, which effectively captures long-range dependencies, and for their scalability and exceptional performance across diverse applications. However, their high computational cost and time requirements limit their adoption in resource-constrained environments. Knowledge distillation provides a straightforward yet effective approach for compressing traditional network architectures, such as CNNs, by transferring knowledge from a large teacher model to a smaller student model. This paper explores the potential for model compression by combining novel Transformer architectures with knowledge distillation. Specifically, we categorize, summarize, and present the substantial body of work on Transformer knowledge and distillation schemes. Furthermore, this survey extends its coverage to widely used knowledge-distillation pipelines for large language models (LLMs) and systematically analyses six major distillation targets. We also review popular applications in various domains, including natural language processing, computer vision, and speech recognition. Finally, we provide forward-looking perspectives, highlighting potential directions for future research on developing highly efficient yet performance-competitive Transformer models through knowledge distillation techniques.