Subset selection for domain adaptive pre-training of language model.

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Pre-trained language models (PLMs) significantly enhance natural language understanding (NLU).
• Domain-adaptive PLMs excel in specific fields but require extensive data and computational resources for training.
• Efficient pre-training methods are needed to reduce the computational burden.

Purpose of the Study:

• To propose an efficient method for pre-training domain-adaptive language models.
• To introduce AlignSet, a novel subset selection technique for optimizing pre-training data.
• To enable faster and more resource-efficient learning of language models.

Main Methods:

• Developed AlignSet, a novel subset selection method.
• Extracted informative subsets from domain-specific datasets for pre-training.
• Conducted experiments across multiple domains to evaluate subset performance.

Main Results:

• AlignSet successfully identifies informative data subsets.
• Subsets generated by AlignSet facilitate faster language model learning compared to using entire datasets.
• Experimental results demonstrate the superiority of AlignSet over existing methods.

Conclusions:

• AlignSet offers an efficient approach to pre-training domain-adaptive language models.
• The method reduces the need for large computational budgets.
• AlignSet enables effective model training with curated, informative data subsets.