RadBERT: Adapting Transformer-based Language Models to Radiology

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Medical Informatics

Background:

• Transformer-based language models (e.g., BERT) are increasingly used in medical applications.
• Adapting general language models to specific domains like radiology can potentially improve performance.
• Radiology natural language processing (NLP) involves extracting meaningful information from unstructured radiology reports.

Purpose of the Study:

• To investigate the efficacy of tailoring transformer-based language models for radiology NLP applications.
• To develop and evaluate a family of radiology-adapted BERT models, named RadBERT.

Main Methods:

• Developed six RadBERT variants by pretraining transformer models on 2.16-4.42 million radiology reports.
• Fine-tuned RadBERT variants on three NLP tasks: abnormal sentence classification, report coding, and report summarization.
• Compared RadBERT performance against five established transformer models using bootstrap resampling.

Main Results:

• RadBERT variants significantly outperformed baseline models in abnormal sentence classification, especially with limited training data (<10%).
• All RadBERT variants showed significant improvements in report coding across five coding systems.
• RadBERT-BioMed-RoBERTa achieved the best performance in report summarization (ROUGE-1 score of 16.18 vs. 15.27).

Conclusions:

• Transformer models specifically tailored to radiology demonstrate superior performance on radiology NLP tasks compared to general models.
• RadBERT models offer a promising approach for advancing automated analysis of radiology reports.
• Domain-specific pretraining enhances the effectiveness of transformer models in specialized medical fields.