The Number Needed to Remind: a Measure for Assessing CDS Effectiveness

Clinical decision support (CDS) is associated with improvement in quality and efficiency in healthcare delivery. The appropriate way to evaluate its effectiveness remains uncertain.

We analyzed data from our electronic health record (EHR) measuring the display frequency of eight reminders for Coronary Artery disease and Type 2 Diabetes and their associated performance according to a predefined methodology. We propose two key performance indicators to measure their impact on a target population: the reminder performance (RP), and the number needed to remind (NNR), to evaluate the impact that Clinical decision support reminders have on the adherence to guideline derived CDS interventions on the entire patient population, and individual providers receiving the interventions.

Data were available for 116,027 patients and a total of 1,982,735 reminders were displayed to a subset of 65,516 patients during the study period from January 1 to December 31, 2010. The evaluation framework assessed provider acknowledgement of the CDS intervention, and the presence of the expected performance event while accounting for patients' exposure to the CDS reminders. The total RP was 2.7% while the average NNR was 3.1 for all the reminders under study.

The proposed framework to asses of CDS performance provides a novel approach to improve the design and evaluation of CDS interventions. The application of this methodology represents an indicator to understand the impact of CDS interventions and subsequent patient outcomes. Further research is required to evaluate the impact of these systems on the quality of care.