Ground truth generalizability affects performance of the artificial intelligence model in automated vertebral...

Computer-aided diagnosis with artificial intelligence (AI) has been used clinically, and ground truth generalizability is important for AI performance in medical image analyses. The AI model was trained on one specific group of older adults (aged≧60) has not yet been shown to work equally well in a younger adult group (aged 18-59).

To compare the performance of the developed AI model with ensemble method trained with the ground truth for those aged 60 years or older in identifying vertebral fractures (VFs) on plain lateral radiographs of spine (PLRS) between younger and older adult populations.

Retrospective analysis of PLRS in a single medical institution.

Accuracy, sensitivity, specificity, and interobserver reliability (kappa value) were used to compare diagnostic performance of the AI model and subspecialists' consensus between the two groups.

Between January 2016 and December 2018, the ground truth of 941 patients (one PLRS per person) aged 60 years and older with 1101 VFs and 6358 normal vertebrae was used to set up the AI model. The framework of the developed AI model includes: object detection with You Only Look Once Version 3 (YOLOv3) at T0-L5 levels in the PLRS, data pre-preprocessing with image-size and quality processing, and AI ensemble model (ResNet34, DenseNet121, and DenseNet201) for identifying or grading VFs. The reported overall accuracy, sensitivity and specificity were 92%, 91% and 93%, respectively, and external validation was also performed. Thereafter, patients diagnosed as VFs and treated in our institution during October 2019 to August 2020 were the study group regardless of age. In total, 258 patients (339 VFs and 1725 normal vertebrae) in the older adult population (mean age 78±10.4; range, 60-106) were enrolled. In the younger adult population (mean age 36±9.43; range, 20-49), 106 patients (120 VFs and 728 normal vertebrae) were enrolled. After identification and grading of VFs based on the Genant method with consensus between two subspecialists', VFs in each PLRS with human labels were defined as the testing dataset. The corresponding CT or MRI scan was used for labeling in the PLRS. The bootstrap method was applied to the testing dataset.

The model for clinical application, Digital Imaging and Communications in Medicine (DICOM) format, is uploaded directly (available at: http://140.113.114.104/vght_demo/svf-model (grading) and http://140.113.114.104/vght demo/svf-model2 (labeling). Overall accuracy, sensitivity and specificity in the older adult population were 93.36% (95% CI 93.34%-93.38%), 88.97% (95% CI 88.59%-88.99%) and 94.26% (95% CI 94.23%-94.29%), respectively. Overall accuracy, sensitivity and specificity in the younger adult population were 93.75% (95% CI 93.7%-93.8%), 65.00% (95% CI 64.33%-65.67%) and 98.49% (95% CI 98.45%-98.52%), respectively. Accuracy reached 100% in VFs grading once the VFs were labeled accurately. The unique pattern of limbus-like VFs, 43 (35.8%) were investigated only in the younger adult population. If limbus-like VFs from the dataset were not included, the accuracy increased from 93.75% (95% CI 93.70%-93.80%) to 95.78% (95% CI 95.73%-95.82%), sensitivity increased from 65.00% (95% CI 64.33%-65.67%) to 70.13% (95% CI 68.98%-71.27%) and specificity remained unchanged at 98.49% (95% CI 98.45%-98.52%), respectively. The main causes of false negative results in older adults were patients' lung markings, diaphragm or bowel airs (37%, n=14) followed by type I fracture (29%, n=11). The main causes of false negatives in younger adults were limbus-like VFs (45%, n=19), followed by type I fracture (26%, n=11). The overall kappa between AI discrimination and subspecialists' consensus in the older and younger adult populations were 0.77 (95% CI, 0.733-0.805) and 0.72 (95% CI, 0.6524-0.80), respectively.

The developed VF-identifying AI ensemble model based on ground truth of older adults achieved better performance in identifying VFs in older adults and non-fractured thoracic and lumbar vertebrae in the younger adults. Different age distribution may have potential disease diversity and implicate the effect of ground truth generalizability on the AI model performance.