Multi-Institutional Evaluation and Training of Breast Density Classification AI Algorithm Using ACR Connect and...

Area of Science:

• Medical imaging analysis
• Artificial intelligence in healthcare
• Breast cancer screening

Background:

• ACR Connect and AI-LAB are software platforms for AI implementation.
• Multi-institutional validation is crucial for robust AI performance.

Purpose of the Study:

• To demonstrate and test the ACR Connect and AI-LAB platform capabilities.
• To implement multi-institutional AI training and validation for breast density classification.

Main Methods:

• Proof-of-concept study involving six US hospitals.
• Installation of Connect and AI-LAB software.
• Training and testing a breast density algorithm on retrospective mammograms.
• Recording implementation timelines (IRB approval, software installation, training/testing).
• Comparing algorithm performance across hospitals and against multi-institutional datasets.

Main Results:

• Median IRB approval time: 66 days; median installation time: 157 days; median training/testing time: 216 days.
• The breast density algorithm performed worse at individual hospitals than on the holdout test dataset, indicating poor generalizability.
• Locally fine-tuned models showed mixed performance and poor results on the test dataset.

Conclusions:

• Successful installation and implementation of Connect and AI-LAB platforms were demonstrated.
• The study highlights the poor generalizability of single-dataset trained algorithms and institution-specific fine-tuned models.
• Emphasizes the importance of multi-institutional testing and training for AI in medical imaging.