External validity of deep learning solution for spontaneous intracranial hemorrhage detection on head CT scans

We externally validated the performance of deep learning (DL) solution for detection of spontaneous intracerebral (ICH), intraventricular (IVH) and subarachnoid hemorrhages (SAH) on non-contrast enhanced head CT scans (NCCTs). We analyzed 901 NCCTs collected retrospectively from two Swiss hospitals: University Hospital Zürich (USZ) and HOCH Health Ostschweiz, Kantonsspital St. Gallen (KSSG). Of these 901 NCCTs, 81 had spontaneous ICH, IVH or SAH. The diagnostic accuracy was evaluated using the radiologist's reports as the reference standard. The DL solution correctly identified 74 out of the 81 intracranial hemorrhages (sensitivity 91.4%). In the USZ cohort, the sensitivity was 88.5% and the specificity was 89.7%. Using the original KSSG-NCCTs, the DL solution had a sensitivity of 100.0% and a specificity of 47.5%. After adjusting the KSSG-NCCTs pixel matrix to a standardized 512 × 512 resolution the KSSG cohort sensitivity remained at 100.0%, and the specificity increased to 74.0%. The overall specificity was 78.4% when using the original imaging data. The overall specificity increased to 85.5% when using the padded KSSG-NCCTs along with original imaging data from USZ. Particularly the specificity varied substantially depending on imaging acquisition parameters. In a clinical setting, this would mean a high variability in a false positive rate.