An Empirical Study on Adaptive Inference for Pretrained Language Model.

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Machine Learning

Background:

• Adaptive inference enhances inference speed for Bidirectional Encoder Representations from Transformers (BERT) with minimal accuracy loss.
• Previous research has primarily focused on BERT, neglecting its application in other Pretrained Language Models (PLMs).

Purpose of the Study:

• To empirically investigate the effectiveness of adaptive inference across diverse PLMs.
• To evaluate the performance of adaptive inference on both English and Chinese language benchmarks.
• To address limitations of adaptive inference, such as its performance with numerous labels.

Main Methods:

• Applied adaptive inference to various PLMs, including Generative Pretraining (GPT), GCNN, ALBERT, and TinyBERT.
• Tested the mechanism on English and Chinese datasets with varying speed thresholds.
• Introduced a label reduction technique to overcome challenges with excessive labels.
• Integrated adaptive inference capabilities into an open-source toolkit, FastPLM.

Main Results:

• Adaptive inference achieved speedups ranging from 1x to 10x across different PLMs and speed thresholds.
• Demonstrated that adaptive inference is compatible with parameter sharing in ALBERT, enabling simultaneous model compression and acceleration.
• Showed that adaptive inference can further accelerate distilled models like TinyBERT.
• Validated the effectiveness of the proposed label reduction method.

Conclusions:

• Adaptive inference is a versatile mechanism applicable to a wide range of PLMs, offering significant speed improvements.
• The FastPLM toolkit simplifies the integration of adaptive inference into practical applications.
• Adaptive inference, combined with techniques like parameter sharing and label reduction, offers a powerful approach for efficient NLP model deployment.