LAD: Layer-Wise Adaptive Distillation for BERT Model Compression.

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Artificial Intelligence

Background:

• Large-scale pre-trained language models (e.g., BERT) show great promise in NLP.
• Their substantial size limits deployment on resource-constrained IoT and edge devices.
• Existing knowledge distillation methods struggle to effectively compress models by reducing layers.

Purpose of the Study:

• To introduce a novel task-specific distillation framework, Layer-wise Adaptive Distillation (LAD).
• To enable effective knowledge transfer from large teacher models to smaller student models with fewer layers.
• To address the lack of sound strategies for distilling knowledge to shallower student models.

Main Methods:

• Developed Layer-wise Adaptive Distillation (LAD), a task-specific distillation framework for BERT model compression.
• Implemented an iterative aggregation mechanism with multiple gate blocks within LAD.
• Designed LAD to adaptively distill layer-wise internal knowledge from teacher to student models without skipping layers.

Main Results:

• LAD successfully enabled effective knowledge transfer to student models with reduced layers.
• Both six-layer and four-layer LAD student models demonstrated superior performance.
• Outperformed previous task-specific distillation approaches on standard GLUE benchmark tasks.

Conclusions:

• Layer-wise Adaptive Distillation (LAD) provides an effective strategy for compressing large pre-trained language models.
• The method facilitates efficient knowledge distillation to shallower student models, suitable for edge deployment.
• LAD significantly improves performance over existing methods for task-specific model compression.