A Serious Flaw in Nutrition Epidemiology: A Meta-Analysis Study.

Area of Science:

• Diet and health research
• Statistical methodology in epidemiology
• Public health nutrition

Background:

• Numerous studies explore diet-health links, with some associating sugar-sweetened beverage (SSB) consumption with Type 2 diabetes.
• Many meta-analyses rely on primary studies that do not adequately adjust for multiple testing or modeling.
• This methodological limitation can render the claims within meta-analyses unreliable due to potential statistical biases in the base studies.

Purpose of the Study:

• To assess the statistical reliability of 10 individual research papers.
• To indirectly evaluate the dependability of a meta-analysis that utilized these 10 papers.
• To investigate the impact of multiple testing and modeling on the validity of health-related dietary claims.

Main Methods:

• Acquired 10 primary research papers used in a meta-analysis.
• Quantified the number of outcomes, predictors, and covariates within each paper.
• Estimated the potential analysis search space (number of comparisons and models) using the formula: outcomes * predictors * 2^covariates, applied to abstract (Space A) and text (Space T) data.

Main Results:

• The median number of possible comparisons varied significantly: 6.5 for Space A (range: 2-12,288) and 196,608 for Space T (range: 3072-117,117,952).
• The median for Space A may be misleading, as individual studies often included 60-165 potential food predictors.
• A large potential analysis search space was identified across the papers, suggesting a high risk of false positives.

Conclusions:

• With a 5% significance level and a vast number of comparisons, nominal statistical significance offers weak support for research claims.
• The statistical claims within the analyzed primary papers are not robustly supported.
• Consequently, the reliability of the meta-analysis drawing conclusions from these papers is compromised.