The P value plot does not provide evidence against air pollution hazards.

Area of Science:

• Environmental Health
• Biostatistics
• Philosophy of Science

Background:

• Epidemiological studies and meta-analyses on air pollution hazards have faced criticism using a graphical method termed the P value plot.
• This method, developed by Young and collaborators, claims to identify zero effects, heterogeneity, and P hacking.
• The P value plot method lacks validation in peer-reviewed literature.

Purpose of the Study:

• To investigate the statistical and evidentiary properties of the P value plot method.
• To assess the validity of claims made using the P value plot regarding air pollution research.

Main Methods:

• A simulation was developed to generate studies and meta-analyses with known true effects.
• The simulation integrated two quantifiable concepts of evidence from philosophy of science.
• The simulation code and analysis are publicly available and reproducible.

Main Results:

• The P value plot did not provide evidence for heterogeneity or P hacking under simulated conditions.
• The plot can indicate zero effects, but only when specific, unmet conditions are present.
• Actual applications of the P value plot by Young and collaborators did not meet these conditions.

Conclusions:

• The P value plot method fails to provide robust evidence for the skeptical claims made by Young and collaborators regarding air pollution hazards.
• The statistical and evidentiary basis of the P value plot is questionable.