P-Hacking Lexical Richness Through Definitions of "Type" and "Token"

Area of Science:

• Biostatistics
• Computational Linguistics
• Biomedical Research Methodology

Background:

• P-hacking, the repeated analysis of data to find statistically significant results, is a known issue in research.
• This practice can lead to spurious findings and inflate the scientific literature with non-replicable results.
• Unintentional p-hacking during data generation remains under-explored.

Purpose of the Study:

• To investigate the potential for p-hacking during the data generation phase.
• To demonstrate how variations in defining key terms can influence statistical outcomes.
• To highlight the implications for interpreting biomedical literature using specific metrics.

Main Methods:

• Utilized the type-token ratio as a case study for analysis.
• Examined how different definitions of "type" and "token" impact results.
• Assessed the frequency of undefined terms in biomedical publications.

Main Results:

• P-hacking was demonstrated to occur during data generation, not solely during analysis.
• Variations in defining "type" and "token" led to significantly different type-token ratio outcomes.
• The study found that these crucial terms are often undefined in the biomedical literature.

Conclusions:

• The ambiguity in defining "type" and "token" for type-token ratio analysis can lead to unintentional p-hacking.
• This lack of definitional clarity hinders the meaningful interpretation of a substantial body of biomedical research.
• Standardized definitions are crucial for reliable and reproducible scientific findings.