Natural Language Processing Model for Identifying Critical Findings-A Multi-Institutional Study.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Radiology

Background:

• Radiology reports are often long and unstructured, hindering timely identification of critical findings and recommendations.
• Effective follow-up of radiology recommendations is a significant clinical challenge.

Purpose of the Study:

• To develop and validate an automated Natural Language Processing (NLP) pipeline for detecting critical findings in radiology reports.
• To compare the performance of a transformer-based ClinicalBERT++ model against a traditional BERT model.

Main Methods:

• Developed an NLP pipeline using a transformer-based ClinicalBERT++ model, fine-tuned on 3 million radiology reports.
• Validated models on internal (EUH) and external (Mayo Clinic) datasets, including different report sections.
• Evaluated performance using precision, recall, and F1-score for critical finding classification.

Main Results:

• ClinicalBERT++ achieved a 0.96 F1-score on an internal test set, matching BERT when using specific sections.
• ClinicalBERT++ outperformed BERT on an external test set, achieving 0.81 precision and 0.54 recall for critical finding reports.
• The model was successfully applied to large-scale radiology reports across five institutions.

Conclusions:

• Automated NLP analysis of radiology reports can identify critical findings and recommendations, enabling automated alerts for clinical follow-up.
• This system acts as a safeguard, reducing overlooked findings and enabling retrospective database analysis.
• The ClinicalBERT++ model demonstrates significant potential for enhancing patient care and clinical workflow in radiology.