Constructing and implementing a performance evaluation indicator set for artificial intelligence decision support...

Area of Science:

• Healthcare Informatics
• Artificial Intelligence in Medicine
• Pediatric Health Systems

Background:

• Artificial intelligence (AI) decision support systems (AI-DSS) are increasingly used in pediatric healthcare.
• Ensuring the performance, interpretability, and continuous monitoring of AI-DSS is critical due to their cost and complexity.
• A systematic approach to evaluating AI-DSS in pediatric settings is needed.

Purpose of the Study:

• To develop a comprehensive and specialized set of evaluation indicators for AI-DSS in pediatric healthcare.
• To establish a framework for continuous performance monitoring and updating of AI-DSS.
• To determine the relative importance (weights) of different performance aspects.

Main Methods:

• A two-stage approach was employed, starting with a literature review, focus group, and Delphi method for indicator development.
• Expert opinions were gathered using the analytic hierarchy process for subjective weights.
• Objective weights were calculated using the entropy weight method, followed by synthesis into combined weights.

Main Results:

• A final indicator set comprised three first-class, fifteen second-class, and forty-seven third-class indicators.
• Organizational performance, societal performance, and user experience performance were identified as key indicator categories.
• Weight analysis revealed 'Organizational performance' as most critical for objective and combined weights, while 'Societal performance' dominated subjective weights.

Conclusions:

• A robust set of evaluation indicators for pediatric AI-DSS has been established and implemented.
• Continuous data collection is necessary for long-term optimization of indicator weights.
• The developed indicators facilitate systematic performance assessment and improvement of AI-DSS in pediatric care.