Evaluating replicability in microbiome data

High-throughput sequencing is widely used to study microbial communities. However, choice of laboratory protocol is known to affect the resulting microbiome data, which has an unquantified impact on many comparisons between communities of scientific interest. We propose a novel approach to evaluating replicability in high-dimensional data and apply it to assess the cross-laboratory replicability of signals in microbiome data using the Microbiome Quality Control Project data set. We learn distinctions between samples as measured by a single laboratory and evaluate whether the same distinctions hold in data produced by other laboratories. While most sequencing laboratories can consistently distinguish between samples (median correct classification 87% on genus-level proportion data), these distinctions frequently fail to hold in data from other laboratories (median correct classification 55% across laboratory on genus-level proportion data). As identical samples processed by different laboratories generate substantively different quantitative results, we conclude that 16S sequencing does not reliably resolve differences in human microbiome samples. However, because we observe greater replicability under certain data transformations, our results inform the analysis of microbiome data.