Related Experiment Video
Updated: Jun 20, 2026

Candidate Gene Testing in Clinical Cohort Studies with Multiplexed Genotyping and Mass Spectrometry
Published on: June 21, 2018
Concordance between nutritional cohorts and randomized trials: biological confirmation or statistical consequence? A
Matthew Calkins1,2, Isaac Núñez3, Adrian Soto-Mota4,5
1School of Medicine, Wake Forest University, Winston-Salem, NC, USA.
Background:
Stadelmaier et al. recently reported a pooled ratio of risk ratios (RRR) of 1.00 between population, intervention/exposure, comparator, outcome (PI/ECO) matched randomized controlled trials (RCTs) and cohort studies, interpreting this as evidence that randomized trials "confirm" observational findings in nutrition with "tremendous public health implications." We hypothesized that this apparent agreement is not a validation of concordance, but an expected statistical effect arising from the pooling of null or small effect sizes with large variances rather than pairwise concordance.
Methods:
We re-analyzed the authors' binary dataset (n = 54 pairs) using a permutation framework (B = 10,000) in which cohort studies were randomly reassigned to RCTs, breaking the original pairing (both globally and restricted within exposure-type strata) while preserving marginal distributions.
Results:
Under random pairing, 99.96% of permuted pooled RRRs fell within the authors' reported confidence interval [0.91,1.10], and the observed RRR was statistically indistinguishable from the random-pairing permutation distribution (p = 0.12). A pairwise discrepancy statistic showed no improvement with matching over random pairing (p = 0.21). A precision-weighted statistic was significant (p = 0.008), but weight was highly concentrated: 4 pairs (7% of n) accounted for 54.8% of total weight, and the effective number of pairs was 9.9 (18% of n). Qualitative concordance using the authors' prior criteria was only 13% (7/54 pairs).
Conclusions:
A pooled RRR near 1.00 indicates only that cohort effect sizes are not systematically different from RCT estimates. In this dataset, an RRR near 1 is a result expected under random pairing. Pairwise agreement is not improved by PI/ECO matching for typical comparisons; the significant weighted statistic reflects alignment among a small minority of high-precision pairs, not "most" findings. Pooled RRR lacks resolution as a confirmation metric in fields with small effects and should not be interpreted as evidence of replication; otherwise, it may inappropriately influence evidence synthesis practices.
More Related Videos
Related Concept Videos
Regression Toward the Mean
Clinical Trials
There are four phases in a clinical trial. A phase one...
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
Bioavailability Study Design: Healthy Subjects Versus Patients
Study Designs in Epidemiology
Observational studies are those where the researcher does not intervene but rather observes natural variations. They include cross-sectional, cohort, and case-control studies.
Bias in Epidemiological Studies

