A Transformer-Based Pipeline for German Clinical Document De-Identification.

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Machine Learning

Background:

• Commercially available large language models (LLMs) like ChatGPT are unsuitable for real patient data due to privacy concerns.
• Manual de-identification of unstructured clinical data is labor-intensive and time-consuming.
• Transformer models offer efficient text processing capabilities for analyzing large datasets.

Purpose of the Study:

• To investigate the impact of large training datasets on the performance of transformer models for de-identifying clinical text.
• To develop and evaluate transformer-based models for detecting sensitive information in German clinical documents.

Main Methods:

• Utilized a training dataset of 10,240 German hospital documents from 1,130 patients.
• Fine-tuned and trained an ensemble of two transformer-based language models.
• Developed annotation guidelines for training annotators and ensuring data consistency.

Main Results:

• The fine-tuned German ELECTRA (gELECTRA) model achieved an F1 macro average score of 0.95 on a test set of 100 documents.
• The gELECTRA model surpassed human annotators, who achieved an F1 score of 0.93.
• The study demonstrated the effectiveness of transformer models in identifying sensitive data within pathology reports and progress notes.

Conclusions:

• Transformer models can be effectively trained to detect sensitive information in real-world German clinical reports.
• An annotation scheme tailored to specific hospital data and comprehensive guidelines improve model performance.
• The best-performing transformer model demonstrated superior accuracy compared to experienced human annotators in de-identifying clinical text.