AI Solution for Identifying Actionable Abnormalities on Chest Radiographs: A Retrospective Case-Control Study

To evaluate the diagnostic performance of a commercial artificial intelligence (AI) solution in detecting actionable abnormalities (AAs) on chest radiographs (CRs) and to compare its performance with that of thoracic radiologists.

Among 986,016 CRs performed at a single academic hospital between 2016 and 2021, 194 CRs were retrospectively identified as actionable based on critical value reports. Age- and sex-matched 388 normal CRs were selected as controls. A commercial AI solution capable of detecting 10 thoracic abnormalities was retrospectively applied to all CRs. Three thoracic radiologists independently reviewed the same dataset. Diagnostic performance was evaluated using the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, positive rate, precision, F1 score, detection yield, and false-referral rate (FRR). Interobserver agreement and agreement between the AI and the radiologists were also assessed.

The AI solution demonstrated excellent diagnostic performance, with an AUC of 0.958, a sensitivity of 90.7%, and a specificity of 91.2%. Compared with the radiologists, the AI showed higher sensitivity and F1 score and achieved the highest detection yield (30.2%, p ≤ 0.02). The FRR of the AI (5.8%) was higher than that of one radiologist (1.7%) but comparable to those of the others. The AI showed substantial agreement (κ = 0.704) with the radiologists' consensus, while interobserver agreement among the radiologists was moderate (κ = 0.45).

The AI solution demonstrated diagnostic performance comparable to that of thoracic radiologists and may assist in detecting AAs when false-positive alerts are appropriately managed.