Critical assessment of transformer-based AI models for German clinical notes

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Bioinformatics

Background:

• Clinical notes are unstructured, limiting their use in health economics and personalized care.
• Deep learning, especially transformer models like BERT, shows potential for structuring clinical data.
• Existing German language models lack biomedical domain adaptation.

Purpose of the Study:

• Evaluate transformer-based models for German biomedical and clinical natural language processing (NLP).
• Compare existing general-purpose models with newly trained biomedical-specific models.
• Assess the effectiveness of domain adaptation for German clinical text.

Main Methods:

• Utilized 8 transformer-based models, pre-training 3 new models on a biomedical corpus.
• Annotated a new clinical notes dataset and used 4 additional corpora.
• Performed named entity recognition (NER) and document classification tasks.

Main Results:

• General-purpose models performed well on clinical NLP tasks.
• The novel BioGottBERT model outperformed GottBERT in clinical NER.
• Training new biomedical models from scratch was not effective.

Conclusions:

• General-purpose language models are suitable for German biomedical NLP tasks.
• Domain adaptation shows potential but is limited by data availability.
• Further development with larger corpora could enhance model performance.