Localizing in-domain adaptation of transformer-based biomedical language models

Area of Science:

• Computational linguistics
• Biomedical informatics
• Natural language processing

Background:

• Digital healthcare generates vast textual data, valuable for improving patient care.
• Fine-tuning language models on domain-specific resources enhances performance.
• Resources for adapting models to less-resourced languages like Italian are scarce.

Purpose of the Study:

• To investigate accessible methods for creating Italian biomedical language models.
• To compare the impact of data quantity versus quality in model adaptation.
• To address the gap in in-domain adaptation for non-English medical resources.

Main Methods:

• Two approaches were explored: neural machine translation of English resources (quantity-focused) and using a native Italian corpus (quality-focused).
• Models were fine-tuned using these distinct datasets.
• Performance was evaluated to determine the effectiveness of each approach.

Main Results:

• Data quantity proved to be a more significant constraint than data quality for biomedical adaptation.
• Concatenating high-quality data enhanced model performance, even with smaller corpora.
• The developed models offer potential for Italian medical research and institutions.

Conclusions:

• Accessible methods can be developed for creating biomedical language models in less-resourced languages.
• Balancing data quantity and quality is key for effective domain adaptation.
• The findings provide insights for generalizing biomedical language model creation to other languages and domains.