Threshold optimization in AI chest radiography analysis: integrating real-world data and clinical subgroups

Area of Science:

• Medical Imaging AI
• Radiology Workflow Optimization
• Clinical Decision Support Systems

Background:

• Manufacturer-defined AI thresholds for chest x-rays (CXRs) often lack customization.
• Optimization strategies using real-world clinical data and pathology-enriched validation data can address subgroup-specific needs.

Purpose of the Study:

• To develop and evaluate a method for optimizing AI thresholds for CXR analysis.
• To tailor AI performance to specific clinical subgroups (inpatient vs. outpatient) and user needs.

Main Methods:

• An AI system and radiologists analyzed a pathology-enriched dataset (563 CXRs) and a routine dataset (15,786 CXRs).
• Iterative receiver operating characteristic analysis linked sensitivity to AI alert rates.
• "Optimized" thresholds (OTs) were defined by a 1% sensitivity increase leading to >1% rise in AI alert rates.

Main Results:

• Optimized thresholds (OTs) varied for inpatients and outpatients, outperforming AI vendor's default thresholds (AIDTs).
• OTs significantly improved sensitivity for pleural effusions and consolidations.
• For inpatient nodule detection, increasing thresholds improved accuracy without compromising sensitivity.

Conclusions:

• A novel, automated method for subgroup-specific AI threshold optimization is proposed.
• This approach is transferable to other AI algorithms and clinical settings.
• Customizing AI thresholds enhances diagnostic accuracy and clinical acceptance of AI tools.