The Number Needed to Remind: a Measure for Assessing CDS Effectiveness

Area of Science:

• Health Informatics
• Clinical Decision Support Systems
• Healthcare Quality Improvement

Background:

• Clinical decision support (CDS) systems can enhance healthcare quality and efficiency, but evaluating their effectiveness is challenging.
• Existing methods for assessing CDS impact are insufficient for comprehensive analysis.
• A robust evaluation framework is needed to understand CDS interventions' real-world performance.

Purpose of the Study:

• To propose and validate a novel framework for evaluating the performance of clinical decision support (CDS) interventions.
• To introduce two key performance indicators: reminder performance (RP) and number needed to remind (NNR).
• To assess the impact of CDS reminders on adherence to guideline-derived interventions in a large patient population.

Main Methods:

• Analysis of electronic health record (EHR) data for 116,027 patients.
• Measurement of display frequency and performance of eight specific CDS reminders for Coronary Artery disease and Type 2 Diabetes.
• Application of a predefined methodology using RP and NNR to evaluate CDS impact on provider and patient levels.

Main Results:

• Over 1.9 million reminders were analyzed for a subset of 65,516 patients.
• The evaluation framework considered provider acknowledgment and performance events, adjusting for patient exposure.
• Overall reminder performance (RP) was 2.7%, with an average number needed to remind (NNR) of 3.1.

Conclusions:

• The proposed framework offers a novel approach to enhance the design and evaluation of CDS interventions.
• The methodology provides valuable indicators for understanding CDS impact on patient outcomes.
• Further research is necessary to fully ascertain the influence of these systems on overall healthcare quality.