Lightweight transformers for clinical natural language processing.

Area of Science:

• Natural Language Processing (NLP)
• Computational Linguistics
• Biomedical Informatics

Background:

• Specialized pre-trained language models show promise in medical NLP.
• Existing models like BioBERT and BioClinicalBERT are often resource-intensive.
• Knowledge distillation enables creation of smaller, efficient models.

Purpose of the Study:

• To develop compact, efficient language models for clinical text processing.
• To create lightweight clinical transformers using knowledge distillation and continual learning.
• To evaluate model performance across various clinical text-mining tasks.

Main Methods:

• Utilized knowledge distillation and continual learning to develop lightweight clinical transformers.
• Ranged model parameter counts from millions to tens of millions.
• Conducted extensive evaluations on standard datasets for multiple NLP tasks.

Main Results:

• Developed efficient, compact clinical transformers with comparable performance to larger models (e.g., BioBERT).
• Achieved superior performance over other compact models trained on general or biomedical data.
• Demonstrated effectiveness across diverse clinical text-mining tasks including NER and relation extraction.

Conclusions:

• This study presents the first comprehensive effort to create efficient and compact transformers for clinical NLP.
• The developed lightweight models offer a viable alternative for resource-constrained clinical text analysis.
• Models and code are publicly available to promote reproducibility and further research.