ACR's Assess-AI: A Registry for Real-World Performance Monitoring of Clinical Imaging Artificial Intelligence

Area of Science:

• Medical Imaging AI
• Health Informatics
• Quality Assurance in Healthcare

Background:

• Clinical imaging artificial intelligence (AI) models require robust post-deployment monitoring.
• Existing frameworks for evaluating real-world AI performance are limited.
• The American College of Radiology (ACR) developed a national registry to address these gaps.

Purpose of the Study:

• To detail the design, infrastructure, and functionality of the ACR's Assess-AI registry.
• To establish a system for monitoring the real-world performance of clinical imaging AI.
• To provide a national quality registry for AI in medical imaging.

Main Methods:

• Assess-AI, part of the National Radiology Data Registry (NRDR), uses ACR Connect for data submission.
• De-identified AI output, radiology reports, and DICOM metadata are processed.
• Large language models (LLMs) extract surrogate labels from reports for AI output comparison.

Main Results:

• The registry supports AI use cases like intracranial hemorrhage and pulmonary embolism detection.
• Facilities can track data completeness, AI performance trends, and compare against benchmarks.
• Interactive dashboards visualize results, and a forensics app allows local review of discordant cases.

Conclusions:

• Assess-AI offers a scalable, privacy-preserving method for monitoring imaging AI.
• It integrates data ingestion, LLM-based labeling, analytics, and local review.
• Future expansions aim to include more modalities and advanced benchmarking for enhanced clinical utility.