The word and the way: strategies for domain-specific BERT pre-training in German medical NLP

Digital healthcare generates vast amounts of clinical texts that hold potential for AI-assisted applications. However, existing German biomedical language models either rely on older architectures or are trained on limited data, which may hinder their performance in real-world settings.

To explore the impact of domain adaptation strategies in German clinical NLP, we developed a family of domain-specific RoBERTa-based language models, collectively referred to as ChristBERT (Clinical- and Healthcare-Related Issues and Subjects Tuned BERT). To address the lack of large-scale German clinical corpora, we curated a 13.5 GB dataset consisting of scientific publications, clinical texts, and health-related web content. Additionally, we employed data augmentation via translation of English clinical corpora. Three domain adaptation strategies were explored: continued pre-training, pre-training from scratch, and pre-training with domain-specific vocabulary adaptation.

The resulting models were evaluated on three medical named entity recognition and two text classification tasks. Our models consistently outperformed four existing general-purpose and medical German models on four out of five tasks. The results demonstrate that the choice of domain adaptation strategy significantly influences downstream task performance. Based on the empirical results, pre-training from scratch is effective for highly specialized clinical texts, whereas continued pre-training is suited for more commonly written medical texts.

ChristBERT establishes a new state-of-the-art for German clinical language modeling. Our findings indicate that the optimal domain adaptation strategy is task-dependent and remains crucial, as adapted models consistently outperformed general-purpose language models in our experiments. To support further research and application in German medical NLP, all developed models are publicly released.