How important is domain-specific language model pretraining and instruction finetuning for biomedical relation...

Area of Science:

• Biomedical Natural Language Processing (NLP)
• Machine Learning
• Computational Linguistics

Background:

• Advances in general NLP, including generative language models (LMs), instruction fine-tuning, and few-shot learning, are increasingly applied to the biomedical domain.
• Domain-specific LMs and fine-tuning methods have been developed for biomedical tasks, aiming to improve performance through specialized training data.

Purpose of the Study:

• To investigate the effectiveness of biomedical-domain language models compared to general-domain models for the task of relation extraction.
• To evaluate the impact of biomedical instruction fine-tuning versus general instruction fine-tuning on model performance for biomedical NLP tasks.

Main Methods:

• Utilized existing language models pretrained on general and biomedical corpora.
• Tested models across four distinct biomedical relation extraction datasets.
• Compared performance of domain-specific LMs against general LMs and evaluated the effect of instruction fine-tuning on various datasets.

Main Results:

• General-domain language models consistently outperformed biomedical-domain models in relation extraction tasks.
• Biomedical instruction fine-tuning significantly improved model performance, achieving results comparable to general instruction fine-tuning.
• Performance gains from biomedical instruction fine-tuning were observed despite using substantially fewer instructions than general fine-tuning datasets.

Conclusions:

• Findings suggest that developing large-scale, domain-specific biomedical LMs may be less effective than focusing on extensive instruction fine-tuning of general LMs for biomedical NLP.
• Future research should prioritize large-scale biomedical instruction fine-tuning of general LMs to enhance performance on downstream tasks like relation extraction.