AI-RADS: A Framework for Assessment of Artificial Intelligence Output in Radiology-Development and Multireader...

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Imaging

Background:

• Growing use of AI in radiology necessitates a standardized reliability assessment framework.
• Current methods lack structured case-level evaluation for AI outputs and overridden reports.

Purpose of the Study:

• To develop and evaluate the Artificial Intelligence Reporting and Data System (AI-RADS).
• AI-RADS aims for objective, case-level assessment of AI reliability, clinical utility, and actions in radiology.

Main Methods:

• A retrospective multireader study involving 5 radiologists evaluating 350 cases.
• Cases were processed by 7 AI applications for image-based and generative tasks.
• Radiologists assigned AI-RADS categories and correctness ratings; interreader agreement quantified using Krippendorff's α.

Main Results:

• Substantial interreader agreement for AI-RADS categories in image-based (α=0.87) and generative AI tasks (α=0.93).
• AI-RADS categories 1-2 aligned with outputs suitable for clinical integration.
• Categories 4-5 correctly identified outputs requiring override or removal.

Conclusions:

• AI-RADS offers a structured framework for evaluating AI reliability and clinical utility in radiology reports.
• The framework demonstrated substantial interreader agreement and broad applicability across AI tools.