Simulating mismatch between calibration and target population in AI for mammography the retrospective VAIB study

Area of Science:

• Artificial intelligence in medical imaging
• Radiology and medical diagnostics

Background:

• Artificial intelligence (AI) models for cancer detection require careful calibration to optimize the balance between cancer detection rate (CDR) and false positive rate.
• Ensuring the reliability of AI tools in clinical settings necessitates understanding how variations in calibration data affect performance.

Purpose of the Study:

• To simulate and quantify the impact of six types of population mismatches on AI cancer detection performance.
• To assess the clinical significance of deviations in CDR caused by non-representative calibration datasets.

Main Methods:

• Simulated non-representative datasets to calibrate AI models for clinical use.
• Introduced mismatches in acquisition year, age, breast density distribution, and mammography vendors between calibration and target populations.
• Evaluated the resulting distortions in cancer detection rate (CDR).

Main Results:

• Mismatching acquisition year caused CDR distortions from -3% to +19%.
• Age mismatches resulted in CDR distortions ranging from -0.2% to +27%.
• Breast density variations led to CDR changes between +1% to 16%.
• Mammography vendor differences introduced significant CDR distortions from -32% to +33%.

Conclusions:

• Mismatches between calibration and target populations introduce clinically significant deviations in AI performance.
• Ensuring calibration datasets are representative of the target clinical population is crucial for safe and effective AI integration in healthcare.