Comparison of Independent Analyses of Identical Image Sets Reveals Significant Analyst-to-Analyst Variability

The Light Microscopy Research Group, a research group with the Association of Biomolecular Resource Facilities, organized a global study where participants were given artificially generated images of various specimens with various signal-to-noise and object proximity levels. Users were tasked with segmenting the images and providing measured metrics as part of the study. Rather than ranking algorithms, our goal was to study the sources of variability of the results across participants given the same task. This study highlights that substantial variability can exist between independent analyses of identical datasets, even when the analysis problem is relatively straightforward and the analysts are experienced. These findings further support the need for data and analysis methods that follow the findable, accessible, interoperable, reusable (FAIR) principles.