Optimizing the Accuracy of Natural Language Processing Tools for Pulmonary Embolism Detection Through Integration...

Area of Science:

• Medical Informatics
• Clinical Natural Language Processing

Background:

• Rule-based natural language processing (NLP) shows promise for identifying pulmonary embolism (PE) in radiology reports.
• The external validity and optimal application of these NLP tools in diverse clinical settings require further investigation.

Purpose of the Study:

• To evaluate the external validity of two NLP algorithms for PE detection in a large, multi-hospital health system.
• To compare the performance of NLP when applied to all radiology reports versus when limited by administrative data (ICD-10 codes and Present-on-Admission indicators).

Main Methods:

• A cross-sectional study analyzed 1,712 hospitalized patients with and without PE at Mass General Brigham hospitals (2016-2021).
• Two NLP algorithms were applied to radiology reports, with physician chart review serving as the reference standard.
• Three approaches were tested: (A) NLP on all reports, (B) NLP on reports with PE discharge codes, and (C) NLP on reports with PE discharge codes or a PE Present-on-Admission indicator.

Main Results:

• Approach A (all reports) showed high sensitivity but low positive predictive value (PPV).
• Approach B (discharge codes) improved PPV but decreased sensitivity.
• Approach C (discharge codes + POA indicator) achieved high sensitivity and PPV, yielding the best F1 scores (88.6%, 94.4%) for both NLP tools, significantly outperforming Approaches A and B (P<0.001).

Conclusions:

• Integrating administrative claims data, specifically ICD-10 codes and Present-on-Admission indicators, with NLP analysis of radiology reports significantly enhances the accuracy of pulmonary embolism detection.
• This combined approach offers a more robust and externally valid method for identifying PE compared to using NLP on radiology reports alone.