Treatment of multiple test readers in diagnostic accuracy systematic reviews-meta-analyses of imaging studies

To evaluate the handling of multiple readers in imaging diagnostic accuracy systematic reviews-meta-analyses.

Search was performed for imaging diagnostic accuracy systematic reviews that performed meta-analysis from 2005-2015. Handling of multiple readers was classified as: 1) averaged; 2) 'best' reader; 3) 'most experienced' reader; 4) each reader counted individually; 5) random; 6) other; 7) not specified. Incidence and reporting of multiple reader data was assessed in primary diagnostic accuracy studies that were included in a random sample of reviews.

Only 28/296 (9.5%) meta-analyses specified how multiple readers were handled: 7/28 averaged results, 2/28 included the best reader, 14/28 treated each reader as a separate data set, 1/28 randomly selected a reader, 4/28 used other methods. Sample of 27/268 'not specified' reviews generated 442 primary studies. 270/442 (61%) primary studies had multiple readers: 164/442 (37%) reported consensus reading, 87/442 (20%) reported inter-observer variability, 9/442 (2%) reported independent datasets for each reader. 26/27 (96%) meta-analyses contained at least one primary study with multiple readers.

Reporting how multiple readers were treated in imaging systematic reviews-meta-analyses is uncommon and method used varied widely. This may result from a lack of guidance, unavailability of appropriate statistical methods for handling multiple readers in meta-analysis, and sub-optimal primary study reporting.