A Two-Phase Framework Leveraging User Feedback and Systemic Validation to Improve Post-Live Clinical Decision Support

Area of Science:

• Health Informatics
• Medical Error Management
• Healthcare Quality Improvement

Background:

• Clinical Decision Support (CDS) systems offer benefits but face risks from malfunctions.
• Evaluating post-live CDS performance in dynamic healthcare environments lacks standardized methods.
• Current user perspectives on CDS performance remain limited.

Purpose of the Study:

• To propose a comprehensive framework for identifying and evaluating post-live CDS malfunctions.
• To assess CDS performance from the perspective of healthcare settings.
• To provide objective insights into CDS issues and improve system performance.

Main Methods:

• A two-phase framework was developed: real-time user feedback and systematic database validation.
• Systematic validation included data flow, knowledge, and rules validation.
• Measures included identity, completeness, plausibility, and consistency across locations and time.

Main Results:

• Seven types of CDS malfunctions were identified over a 2-year period in 14 Canadian acute care facilities.
• The overall malfunction rate was less than 2%.
• Increased malfunctions were observed during electronic health record (EHR) upgrade and implementation phases.

Conclusions:

• The proposed framework enables comprehensive evaluation of CDS performance in healthcare settings.
• It offers objective insights into malfunction prevalence, including low-frequency issues.
• Applying this framework can enhance CDS reliability and user trust.