Shaping pre-trained language models for task-specific embedding generation via consistency calibration.

Area of Science:

• Natural Language Processing
• Machine Learning

Background:

• Pre-trained language models (PLMs) are crucial for downstream tasks but can forget pre-trained knowledge during fine-tuning.
• This knowledge forgetting limits performance on specific tasks.

Purpose of the Study:

• To introduce EGO-PLM, a novel approach that uses PLMs as task-specific embedding generators.
• To align fine-tuning tasks with pre-training tasks to mitigate knowledge forgetting.

Main Methods:

• EGO-PLM employs a task-agnostic pre-defined task similar to pre-training.
• A task-specific embedding generator adapts PLMs to downstream tasks, trained jointly with the pre-defined task.
• Consistency calibration (CoCa) uses adversarial training to align pre-defined and task-specific objectives, resolving conflicts and ensuring task-specific embeddings.

Main Results:

• EGO-PLM demonstrated consistent and substantial improvements across 8 datasets and 6 task categories.
• The approach outperformed state-of-the-art baselines in fine-tuning performance.

Conclusions:

• EGO-PLM effectively addresses the challenge of knowledge forgetting in PLMs during fine-tuning.
• The proposed method enhances downstream task performance by leveraging PLMs as task-specific embedding generators.