Related Experiment Video
Updated: Aug 10, 2026
An R-Based Landscape Validation of a Competing Risk Model
Published on: September 16, 2022
External validation of a top-ranked model from the RSNA pulmonary embolism detection challenge: assessment of
Patricia Hahlbohm1, Sebastian Starke2, Felix Schön3
1Institute and Polyclinic for Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Carl Gustav Carus Dresden, TUD Dresden University of Technology, Fetscherstraße 74, 01307, Dresden, Germany. patricia.hahlbohm@ukdd.de.
Abstract:
To externally validate the RSNA 2020 challenge 2nd-place deep-learning (DL) algorithm for detecting pulmonary embolism (PE) on computed tomography pulmonary angiography (CTPA) scans with additional focus on subsegmental-only PEs (SSPE). Between 2015 and 2022 1,038 CTPAs were retrospectively enrolled. An experienced radiologist (> five years) labelled CTPAs for the presence, location (central, lobar and segmental, subsegmental-only) and side (right-sided, left-sided, bilateral) of PE. The DL model was tested for its ability to predict these labels by analyzing accuracy, sensitivity, specificity and area under the receiver operating characteristic curve (AUROC) using different model outcome probabilities for each analysis. Overall, 136 central, 375 peripheral PEs (303 lobar and segmental PEs, 72 SSPEs), and 527 patients without any PE were analyzed. The model correctly predicted the presence of any PE in 921/1,038 patients (88.7%), yielding a sensitivity of 0.80, specificity of 0.97 and AUROC of 0.94. No central PE was missed, whereas 100/375 (26.7%) peripheral PEs remained undetected. Using the corresponding model output probability, the central PE status was correctly identified in 999/1,038 (96.2%) patients, while peripheral PE was correctly identified in 785/1,038 (75.6%) patients. This corresponded to sensitivities of 0.94 and 0.77, specificities of 0.97 and 0.75, and AUROC values of 0.99 and 0.74, respectively. The model performed better in detecting right-sided compared to left-sided PEs (AUROC: right- vs. left-sided: 0.95 and 0.92, p < 0.05). SSPE status was correctly predicted in 796/1,038 (76.7%) patients, yielding a sensitivity of 0.94, a specificity of 0.22, and an AUROC of 0.48, respectively. The model demonstrated high performance in detecting any PE, performing best on central and slightly lower on lobar and segmental PE. Predicting the presence of SSPEs was difficult, likely due to the model not being explicitly trained for this subtype.