Domain Word Extension Using Curriculum Learning.

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Self-supervised learning models like BERT excel in NLP tasks but struggle with domain-specific vocabulary.
• Retraining models for specialized domains is data-intensive and time-consuming.
• Adapting general models to new domains without extensive retraining is a significant challenge.

Purpose of the Study:

• To propose an efficient method for adapting pre-trained language models to specific domain vocabularies.
• To enable quick and effective application of general domain models to specialized tasks.
• To overcome the limitations of out-of-domain performance degradation in NLP models.

Main Methods:

• Extracting meaningful wordpieces from downstream task training data to extend vocabulary.
• Utilizing curriculum learning with successive model updates to adapt new vocabulary embeddings.
• Integrating vocabulary adaptation within a single training run for downstream tasks.

Main Results:

• Demonstrated stable performance improvements on Korean classification tasks (AIDA-SC, AIDA-FC, KLUE-TC).
• Successfully adapted pre-trained models to specific domain vocabularies without full retraining.
• Validated the effectiveness of the proposed curriculum learning approach for embedding adaptation.

Conclusions:

• The proposed method offers a convenient and effective solution for domain adaptation in NLP.
• This approach reduces the need for extensive data and computational resources for specialized model training.
• The technique facilitates broader application of powerful pre-trained language models across diverse domains.