Language model-based labeling of German thoracic radiology reports

Area of Science:

• Medical Informatics
• Artificial Intelligence in Radiology
• Natural Language Processing

Background:

• Accurate labeling of thoracic radiology reports is crucial for training diagnostic models.
• Manual annotation is time-consuming and expensive, necessitating automated solutions.
• Weak supervision offers a promising avenue for improving label extraction efficiency.

Purpose of the Study:

• To develop and evaluate a deep learning-based label prediction model for German free-text thoracic radiology reports.
• To assess the model's performance in extracting CheXpert labels and its utility in training chest X-ray classification models.
• To compare the efficacy of manual annotations, rule-based labels, and deep learning-based labels for training diagnostic models.

Main Methods:

• A German BERT encoder was utilized for a label extraction model, trained on manual annotations, rule-based labels, and a combination thereof.
• Label extraction performance was quantified using F1 scores for mention extraction, negation detection, and uncertainty detection.
• Chest X-ray classification models (DenseNet-121) were trained using different label sources to evaluate their impact on pneumothorax detection.

Main Results:

• The deep learning labeler (DL) significantly outperformed the rule-based labeler (RB) across all label extraction tasks (mention, negation, uncertainty) on manually labeled data.
• Models pre-trained on a larger dataset (DS 0) and fine-tuned on a smaller labeled set (DS 1) showed superior performance.
• Chest X-ray pneumothorax classification achieved the highest performance (AUC 0.939) when trained with DL-predicted labels, surpassing RB labels (AUC 0.858) and closely matching manual labels (AUC 0.934).

Conclusions:

• Weak supervision, particularly with a rule-based report labeler, enhances labeling performance for deep learning models.
• The proposed deep learning-based label extraction model serves as an effective substitute for extensive manual labeling, requiring minimal annotated data.
• Training diagnostic models with predicted labels demonstrates comparable performance to using manually annotated data, highlighting the potential of automated labeling in medical imaging.