PROACTIVE-AI: An Intelligent Dashboard for Assessing and Predicting of FDA-approved AI Software Performance in...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Regulatory Science

Background:

• The increasing number of FDA-approved AI tools in healthcare brings significant benefits but also ethical, regulatory, and safety challenges.
• Existing regulatory pathways have gaps, including inconsistent pre-market evaluations and insufficient post-market surveillance of AI devices.
• Lack of real-world performance data and accountability frameworks hinders trust in AI medical devices.

Purpose of the Study:

• To develop an interactive dashboard, PROACTIVE-AI, for assessing and predicting the real-world performance of FDA-approved AI software.
• To provide structured guidance on the anticipated performance of AI-enabled medical devices.
• To identify device characteristics and contextual factors linked to elevated deployment risks.

Main Methods:

• Utilized publicly available FDA data to develop the PROACTIVE-AI dashboard.
• Incorporated knowledge graph visualization and longitudinal trend monitoring of performance indicators (e.g., recalls, safety issues).
• Developed an AI-aided post-market surveillance risk assessment calculator based on historical recall data.

Main Results:

• PROACTIVE-AI enables exploratory analysis of AI medical device performance and safety.
• The dashboard highlights challenges in real-world monitoring and accountability of deployed AI.
• Identified key device characteristics and contextual factors associated with increased deployment risk.

Conclusions:

• PROACTIVE-AI demonstrates the potential to narrow the trust gap in AI healthcare by offering quantitative performance metrics.
• The dashboard aids stakeholders in understanding expected clinical performance and recall risks.
• Emphasizes the need for robust post-market surveillance and accountability for AI medical devices.