Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM).

Area of Science:

• Radiology
• Medical Imaging
• Artificial Intelligence

Background:

• Commercial artificial intelligence (AI) tools in radiology often lack real-time monitoring after deployment.
• The absence of confidence assessments for AI predictions increases clinician cognitive load and risk of misdiagnosis.
• Existing AI monitoring methods may not be suitable for "black-box" commercial products.

Purpose of the Study:

• To introduce the Ensembled Monitoring Model (EMM), a novel framework for monitoring AI performance in radiology.
• To provide robust confidence measurements for AI predictions without accessing internal AI components.
• To reduce cognitive burden and improve the reliability of AI tools in clinical practice.

Main Methods:

• Developed EMM, a framework inspired by clinical consensus and multiple expert reviews.
• EMM operates independently on AI outputs, suitable for black-box commercial AI.
• Tested EMM on a dataset of 2919 intracranial hemorrhage detection studies.

Main Results:

• EMM successfully categorized confidence levels for AI-generated predictions.
• The framework demonstrated the potential to improve overall AI tool performance.
• EMM suggested different clinical actions based on confidence assessments, aiding clinicians.

Conclusions:

• EMM offers a viable solution for monitoring deployed AI tools in radiology.
• The framework enhances AI reliability and reduces clinician cognitive burden.
• Key technical considerations and best practices for clinical translation of EMM are provided.