Cost-Efficient Domain-Adaptive Pretraining of Language Models for Optoelectronics Applications

Area of Science:

• Optoelectronics
• Natural Language Processing (NLP)
• Artificial Intelligence (AI)

Background:

• Pretrained language models (PLMs) show versatility in NLP and applications like data mining in optoelectronics.
• Bidirectional Encoder Representations from Transformers (BERT) is a widely adopted architecture in scientific domains.
• General NLP models may not fully capture the nuances of optoelectronics research literature.

Purpose of the Study:

• To introduce novel optoelectronics-aware BERT models (OE-BERT, OE-ALBERT, OE-RoBERTa).
• To evaluate their performance against general English models and larger counterparts on optoelectronics-related NLP tasks.
• To demonstrate an efficient domain-adaptive pretraining (DAPT) method for scientific NLP.

Main Methods:

• Development of three specialized BERT architectures: OE-BERT, OE-ALBERT, and OE-RoBERTa.
• Domain-adaptive pretraining (DAPT) applied to RoBERTa for optoelectronics text.
• Comparative evaluation of model performance on various optoelectronics NLP tasks.

Main Results:

• OE-BERT, OE-ALBERT, and OE-RoBERTa models surpassed general English BERT models and larger models in optoelectronics NLP tasks.
• The DAPT method for RoBERTa achieved significant computational savings (>80%) in pretraining.
• Performance was maintained or enhanced using the cost-effective DAPT approach.

Conclusions:

• Optoelectronics-specific BERT models offer superior performance for NLP tasks in this domain.
• Domain-adaptive pretraining is an effective and resource-efficient strategy for developing specialized scientific language models.
• The developed models and datasets are publicly available to the optoelectronics research community.