Dataset decay and the problem of sequential analyses on open datasets.

Area of Science:

• Statistics
• Data Science
• Research Methodology

Background:

• Open data initiatives promote data sharing and reuse.
• Multiple statistical testing on the same dataset can lead to inflated Type I error rates (false positives).

Purpose of the Study:

• To demonstrate how sequential hypothesis testing by multiple researchers on a single dataset inflates error rates.
• To discuss potential correction procedures for reducing false positives in such scenarios.

Main Methods:

• Simulation studies to model sequential hypothesis testing on shared datasets.
• Analysis of error rate inflation under various testing scenarios.

Main Results:

• Sequential hypothesis testing significantly increases the probability of false positives.
• The extent of error inflation is dependent on the number of tests and researchers.

Conclusions:

• Researchers must be aware of the increased risk of false positives when reusing open datasets.
• Developing and applying robust correction procedures is crucial for maintaining statistical integrity in data reuse.