Factors Impacting the Performance of Deep Learning Detection of Pulmonary Emboli

Area of Science:

• Radiology
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• AI models are increasingly used in clinical practice.
• Generalizability of AI models outside controlled settings is a concern.
• Real-world performance evaluation of FDA-cleared AI tools is crucial.

Purpose of the Study:

• Evaluate the real-world performance of a commercial AI pulmonary embolism (PE) detection model.
• Identify technical, demographic, and clinical factors influencing AI performance variation.
• Inform post-production monitoring and deployment strategies for AI tools.

Main Methods:

• Retrospective analysis of 11,144 CT pulmonary angiography examinations.
• Commercial PE detection model processed all imaging data.
• Extracted technical parameters, demographics, and comorbidities from DICOM headers and EHRs.
• Utilized univariate and multivariable logistic regression to identify performance-associated factors.

Main Results:

• Overall model sensitivity was 83.5% and positive predictive value (PPV) was 90.5%.
• Decreased sensitivity associated with specific scanner manufacturers, increased slice thickness, and imaging artifacts.
• Patient comorbidities like heart failure and pulmonary hypertension significantly reduced AI sensitivity.
• Demographic factors (age, sex, race, BMI) did not show significant associations with model performance.

Conclusions:

• AI performance in clinical practice is influenced by technical imaging parameters and patient comorbidities.
• Understanding these variables is essential for selecting appropriate AI tools and for effective post-deployment monitoring.
• Local validation frameworks are needed to ensure safe AI deployment across diverse healthcare settings.