Screening p-hackers: Dissemination noise as bait

Area of Science:

• Statistical methodology
• Research integrity

Background:

• Data mining, or p-hacking, generates spurious findings by fitting numerous statistical models.
• Uninformed researchers often fall prey to p-hacking, while informed researchers with hypotheses are less affected.

Purpose of the Study:

• To investigate the effectiveness of adding noise to data as a screening mechanism against p-hacking.
• To analyze how dissemination noise impacts different types of researchers and research credibility.

Main Methods:

• Theoretical analysis of how noise affects researchers with and without prior hypotheses.
• Asymptotic analysis as the number of observations grows large.
• Characterization of optimal noise levels in a tractable special case.

Main Results:

• Dissemination noise effectively screens out p-hacked findings by creating 'baits'.
• Uninformed researchers engaging in data mining are more susceptible to noise than informed researchers.
• As data size increases, noise achieves optimal screening asymptotically.

Conclusions:

• Dissemination noise, a privacy tool, can be repurposed to enhance research credibility by screening p-hackers.
• Optimal noise levels can be determined, balancing screening effectiveness and potential information loss.