Development and Validation of a Sham-AI Model for Intracranial Aneurysm Detection at CT Angiography

Area of Science:

• Medical Imaging
• Artificial Intelligence in Medicine
• Neuroradiology

Background:

• Artificial intelligence (AI) models are increasingly used for medical diagnoses.
• Evaluating the true impact of AI requires rigorous control measures, such as placebo comparisons.
• Intracranial aneurysms are a significant cause of morbidity and mortality, necessitating accurate diagnostic tools.

Purpose of the Study:

• To develop and assess a sham artificial intelligence (AI) model as a placebo control.
• To compare the diagnostic performance of radiologists with and without AI assistance.
• To evaluate the impact of a sham AI on radiologists' diagnostic accuracy for intracranial aneurysms.

Main Methods:

• A retrospective, blinded, multireader, multicase study involving 16,422 CT angiography examinations.
• Development of a sham AI with near-zero sensitivity and standard AI with high sensitivity and specificity.
• Twenty-eight radiologists were assigned to read cases with standard AI, sham AI, or no AI assistance in a crossover design.

Main Results:

• Standard AI demonstrated high sensitivity (96.0%) and specificity (82.0%), while sham AI performed poorly (0.0% sensitivity, 76.0% specificity).
• Radiologists assisted by standard AI showed significantly improved sensitivity (20.7% increase) compared to sham AI-assisted readings.
• Sham AI assistance did not compromise radiologists' diagnostic performance, with sensitivity and specificity comparable to unassisted readings.

Conclusions:

• The developed sham AI model effectively served as a placebo control in evaluating AI-assisted diagnosis.
• Radiologists' diagnostic performance was not negatively affected by the sham AI, indicating its inertness as a control.
• This study validates the methodology for assessing AI tools in medical diagnostics, confirming the benefits of standard AI for intracranial aneurysm detection.