Fine-tuning large neural language models for biomedical natural language processing.

Area of Science:

• Computational Biology
• Natural Language Processing
• Machine Learning

Background:

• Large neural language models have advanced Natural Language Processing (NLP).
• Fine-tuning these models for specific tasks, particularly in biomedical NLP with limited labeled data, presents significant challenges due to increasing model size and potential instability.
• Existing methods often struggle with low-resource scenarios.

Purpose of the Study:

• To systematically investigate fine-tuning stability in biomedical NLP.
• To explore and evaluate techniques for enhancing the performance of large language models in low-resource biomedical applications.
• To establish new state-of-the-art results on various biomedical NLP tasks.

Main Methods:

• Conducted a systematic study on fine-tuning stability for large neural language models in biomedical NLP.
• Explored and compared techniques including freezing lower layers, layerwise decay, and reinitializing top layers.
• Evaluated model performance on low-resource biomedical text similarity tasks, such as BIOSSES.
• Investigated the impact of domain-specific vocabulary and pretraining.

Main Results:

• Fine-tuning performance is sensitive to pretraining settings.
• Freezing lower layers benefits standard BERT models.
• Layerwise decay is more effective for BERT and ELECTRA models.
• Reinitializing top layers is optimal for low-resource text similarity tasks.
• Domain-specific vocabulary and pretraining enhance model robustness.

Conclusions:

• Specific fine-tuning strategies significantly improve performance for low-resource biomedical NLP applications.
• The identified techniques address fine-tuning instability, leading to more robust models.
• This research establishes new state-of-the-art performance across a range of biomedical NLP tasks.