A self-supervised language model selection strategy for biomedical question answering.

Area of Science:

• Natural Language Processing
• Biomedical Informatics
• Machine Learning

Background:

• Pre-trained Language Models (PLMs) excel in Information Retrieval and NLP tasks.
• Domain-specific PLMs offer high performance but are computationally expensive.
• General-purpose PLMs are more accessible but may lack domain specificity.

Purpose of the Study:

• To investigate if general-purpose PLMs can match domain-specific PLM performance without retraining.
• To explore synergistic effects of combining multiple general-purpose PLMs.
• To develop a computationally efficient method for domain-specific tasks.

Main Methods:

• A self-supervised classifier was trained to select the most suitable general-purpose PLM for biomedical questions.
• Experiments were conducted on the BioASQ dataset for biomedical question answering.
• Performance was evaluated by comparing general-purpose PLMs with and without the selection strategy against domain-specific models.

Main Results:

• General-purpose PLMs, when combined using the proposed selection strategy, showed synergistic performance improvements.
• Statistically significant improvements of 16.7% (lighter models) and 14.2% (larger models) were observed.
• The method enabled general-purpose PLMs to achieve performance competitive with domain-specific models like PubMedBERT.

Conclusions:

• Leveraging general-purpose PLMs with a smart selection strategy is a computationally efficient alternative to retraining.
• This approach enhances the utility of readily available PLMs for specialized domains like biomedicine.
• The findings suggest a practical pathway for improving biomedical question answering systems.