The reliability of an environmental epidemiology meta-analysis, a case study

Area of Science:

• Environmental Health
• Epidemiology
• Biostatistics

Background:

• Replication crisis in science, with many research claims failing to reproduce.
• Observational studies, often used in meta-analyses, can be unreliable.
• Meta-analyses are valuable but depend critically on the quality of underlying studies.

Purpose of the Study:

• To assess the statistical reliability of base studies used in an air quality/heart attack meta-analysis.
• To evaluate the validity of the meta-analysis itself based on the quality of its source data.
• To identify potential biases and methodological issues in observational research linking air quality and heart attacks.

Main Methods:

• Examination of a meta-analysis comprising 14 observational air quality/heart attack studies.
• Utilized simple counting to assess the reliability of individual base studies.
• Employed p-value plots to investigate statistical heterogeneity among the base studies.

Main Results:

• Identified potential for extensive multiple testing and modeling without statistical correction in base studies.
• Found that statistics from base papers may not be unbiased, a prerequisite for reliable meta-analysis.
• Detected significant study heterogeneity, with evidence suggesting p-hacking in several studies.

Conclusions:

• The base observational studies exhibit methodological weaknesses, including potential for uncorrected multiple testing and p-hacking.
• These identified issues compromise the reliability of the individual studies.
• Consequently, the meta-analysis derived from these flawed studies is deemed unreliable.