Meta-analysis of the accuracy of tools used for binary classification when the primary studies employ different...

The quality of tools used in binary classification is evaluated by studies that assess the accuracy of the classification. The empirical evidence is summarized in 2 × 2 contingency tables. These provide the joint frequencies between the true status of a sample and the classification made by the test. The accuracy of the test is better estimated in a meta-analysis that synthesizes the results of a set of primary studies. The true status is determined by a reference that ideally is a gold standard, which means that it is error free. However, in psychology, it is rare that all the primary studies have employed the same reference, and often they have used an imperfect reference with suboptimal accuracy instead of an actual gold standard. An imperfect reference biases both the estimates of the accuracy of the test and the empirical prevalence of the target status in the primary studies. We discuss several strategies for meta-analysis when different references are employed. Special attention is paid to the simplest case, where the meta-analyst has 1 group of primary studies using a reference that can be considered a gold standard and a 2nd group of primary studies using an imperfect reference. A procedure is recommended in which the frequencies from the primary studies with the imperfect reference are corrected prior to the meta-analysis itself. Then, a hierarchical meta-analytic model is fitted. An example with actual data from SCOFF (Sick-Control-One-Fat-Food; Hill, Reid, Morgan, & Lacey, 2010; Morgan, Reid, & Lacey, 1999) a simple but efficient test for detecting eating disorders, is described.