The effect of performance indicator category on estimates of intervention effectiveness

A challenge for systematic reviews on improving health worker performance is that included studies often use different performance indicators, and the validity of comparing interventions with different indicators is unclear. One potential solution is to adjust comparisons by indicator category, with categories based on steps of the case-management process that can be easily recognized (assessment of symptoms, treatment etc.) and that might require different levels of effort to bring about improvements. However, this approach would only be useful if intervention effect sizes varied by indicator category. To explore this approach, studies were analyzed that evaluated the Integrated Management of Childhood Illness (IMCI) strategy.

Performance indicators were grouped into four categories: patient assessment, diagnosis, treatment and counseling. An effect size of IMCI was calculated for each indicator. Linear regression modeling was used to test for differences among the mean effect sizes of the indicator categories.

Six studies were included, with data from 3136 ill child consultations. Mean effect sizes for 63 assessment indicators, 12 diagnosis indicators, 31 treatment indicators and 34 counseling indicators were 50.9 percentage-points (%-points), 44.7, 36.5 and 46.6%-points, respectively. After adjusting for baseline indicator value, compared with the assessment mean effect size, the diagnosis mean was 7.3%-points lower (P = 0.23), the treatment mean was 15.2%-points lower (P = 0.0004) and the counseling mean was 12.9%-points lower (P = 0.0027).

Adjusting the results of systematic reviews for indicator category and baseline indicator value might be useful for improving the validity of intervention comparisons.