Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing...

Area of Science:

• Artificial Intelligence in Radiology
• Natural Language Processing in Healthcare
• Machine Learning for Medical Imaging Analysis

Background:

• Low reported rates of recommendations for additional imaging (RAIs) hinder quality improvement efforts in radiology.
• Bidirectional Encoder Representations from Transformers (BERT) offer potential for identifying RAIs due to its language understanding capabilities.

Purpose of the Study:

• To develop and externally validate an artificial intelligence (AI)-based model for identifying radiology reports containing RAIs.
• To assess the performance of a BERT-based model against traditional machine learning (TML) for RAI detection.

Main Methods:

• A retrospective study utilized 6300 radiology reports for training and testing a BERT-based RAI detection model.
• An external validation set of 1260 reports from multiple sites was used to assess model generalizability.
• Radiology reports were manually reviewed by experts to identify the presence of RAIs for model training and evaluation.

Main Results:

• The BERT-based model achieved 96.4% F1 score in the test set, significantly outperforming the TML model (67.2% F1 score).
• In the external validation set, the BERT-based model demonstrated high performance with a 95.2% F1 score and 99.0% accuracy.
• The model accurately identified 10.0% of reports containing RAIs among 7560 analyzed reports.

Conclusions:

• The BERT-based AI model accurately identifies RAIs in radiology reports, surpassing traditional machine learning approaches.
• High performance on external validation suggests the model's adaptability across different healthcare systems without extensive retraining.
• The AI model holds potential for real-time electronic health record monitoring to ensure timely follow-up care.