PET: Parameter-efficient Knowledge Distillation on Transformer.

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Transformer models offer significant performance gains in NLP tasks.
• Large Transformer models present deployment challenges due to size, computational cost, and inference time.
• Existing compression methods often overlook the decoder, which contributes significantly to inference latency.

Purpose of the Study:

• To develop an efficient Transformer compression method that reduces the size of both encoder and decoder.
• To maintain the performance of large Transformer models in a compressed format.
• To enable deployment of Transformer models on resource-constrained devices.

Main Methods:

• Parameter-Efficient knowledge distillation on Transformer (PET) utilizes efficient weight sharing between parameter groups.
• PET incorporates a warm-up process with a simplified task to enhance knowledge distillation gains.
• The method focuses on compressing both the encoder and decoder components of Transformer models.

Main Results:

• PET significantly reduces memory usage and accelerates inference speed compared to uncompressed models.
• On the IWSLT'14 EN→DE machine translation task, PET achieved an 81.20% reduction in memory and a 45.15% increase in inference speed.
• The compression resulted in a minor decrease of only 0.27 in BLEU score, demonstrating performance retention.

Conclusions:

• PET offers an effective solution for compressing large Transformer models for efficient deployment.
• The method outperforms existing Transformer compression techniques, particularly in machine translation.
• PET enables significant efficiency gains without substantial performance degradation, making large models more accessible.