Reducing interrater variability and improving health care: a meta-analytical review

In the scientific literature about reliability, the main approach to increasing reliability seems to involve increasing the number of observers and improving the instrument used. Other aspects for improving reliability - like the training of raters - seem to receive less notice. It is worth asking whether this technical approach could be complemented by training the user of the instrument. A systematic meta-analytical review of the research literature was performed to answer this question and examine the effectiveness of planned interventions for improving interrater reliability of health care professionals.

The databases of PubMed (MEDLINE), Embase, Omega and PsycINFO were searched. The inclusion criteria were met by 57 studies. Details extracted from the studies included the study design, the number of observers and the number of observed cases, the intervention, the type of instrument (whether or not it was highly technical), and statistical information about the agreement before and after the intervention. Interventions were categorized into three groups: training of professionals, improving the diagnostic instrument and a combination of training and improving the instrument. A meta-analysis was performed by means of linear regression.

The interventions were arranged according to their effectiveness in improving the diagnostic instrument (mean change: β = 0.13), training combined with improving the instrument (mean change: β = 0.10) and training (mean change: β = 0.09).

On average, although all types of interventions are effective, improving the diagnostic instrument seems to be the most effective. Especially when highly technical instruments were concerned, improvement proved to be very effective (β = 0.52). Because instrumental variables constitute a major source of error, improving the instrument is an important approach. However, this review offers solid arguments that can complement the literature and practice, with a focus on training the user of the instrument.