Interpreting observational studies: why empirical calibration is needed to correct p-values

Area of Science:

• Pharmacoepidemiology
• Biostatistics
• Medical Research Methodology

Background:

• The common reliance on p < 0.05 in medical literature assumes a 5% chance of false positives.
• Observational studies, unlike randomized trials, are particularly susceptible to bias and confounding, potentially invalidating this assumption.
• The integrity of statistical significance in drug safety research using observational data requires rigorous validation.

Purpose of the Study:

• To assess the validity of the p < 0.05 threshold in observational drug safety studies.
• To evaluate the impact of bias and confounding on statistical significance in real-world data.
• To introduce and test empirical calibration of p-values to account for systematic error.

Main Methods:

• Replication of three exemplar observational drug safety studies (case-control, cohort, self-controlled case series).
• Application of these designs to negative control drugs (drugs not expected to cause the outcome).
• Calculation of calibrated p-values incorporating random and systematic error, followed by literature analysis.

Main Results:

• A high frequency of spurious statistical significance (p < 0.05) was observed when no true effect was present.
• Empirical calibration effectively reduced false positive findings to the nominal 5% level.
• Analysis suggests at least 54% of literature findings with p < 0.05 may not be statistically significant.

Conclusions:

• The standard p < 0.05 threshold is unreliable in many observational drug safety studies.
• Empirical calibration is a vital tool for correcting systematic error and improving the reliability of research findings.
• A significant proportion of published observational study results warrant reevaluation due to potential false positives.