FLAT: Fusing layer representations for more efficient transfer learning in NLP.

Area of Science:

• Natural Language Processing
• Machine Learning

Background:

• Parameter efficient transfer learning (PETL) methods offer efficient fine-tuning of pre-trained language models (PLMs).
• Existing PETL methods often apply uniform structures across all PLM layers and neglect layer-specific knowledge, while incurring high computational and memory costs due to backpropagation through frozen PLMs.

Purpose of the Study:

• To propose FLAT, a novel PETL method designed to individually leverage knowledge from all layers of PLMs.
• To enhance transfer learning efficiency by decoupling the backpropagation path from the frozen PLM.

Main Methods:

• FLAT treats the backbone PLM as a feature extractor, integrating features from all layers within a separate side-network.
• This architecture avoids backpropagation through the frozen PLM, significantly reducing memory requirements.

Main Results:

• FLAT demonstrates superior performance compared to other tuning techniques in low-resource scenarios on the GLUE benchmark.
• It achieves comparable performance in high-resource scenarios while utilizing only 0.53% trainable parameters and reducing GPU memory usage by 3.2× with BERT_base.
• Ablation studies confirm the effectiveness of the proposed fusion layer in integrating PLM knowledge for downstream tasks.

Conclusions:

• FLAT offers a more efficient and effective approach to parameter efficient transfer learning by exploiting layer-wise knowledge diversity.
• The method presents a significant advancement in reducing computational and memory overhead for fine-tuning large language models.