Bias in Zipf's law estimators.

Area of Science:

• Quantitative Linguistics
• Statistical Modeling
• Computational Linguistics

Background:

• Rank-frequency distributions, often modeled by power laws (e.g., Zipf's Law), are common in natural language.
• Existing maximum likelihood estimators (MLEs) for inferring power law exponents from such data are known to be biased.
• The bias in MLEs stems from the use of an inappropriate likelihood function.

Purpose of the Study:

• To derive the correct likelihood function for power law inference from rank-frequency data.
• To investigate the bias of existing and novel estimators, including approximate Bayesian computation (ABC), for power law models.
• To assess the impact of assuming simple probability distributions versus complex natural language processes on bias.

Main Methods:

• Derivation of the theoretically correct likelihood function for power law models.
• Implementation and evaluation of an approximate Bayesian computation (ABC) method.
• Analysis of bias in estimators when applied to both idealized Zipfian distributions and natural language data.

Main Results:

• The correct likelihood function is computationally intractable.
• The proposed ABC method exhibits reduced bias compared to traditional MLEs for idealized data.
• Both traditional MLEs and the investigated ABC method retain significant bias when applied to natural language due to the assumption of simple probability distributions.

Conclusions:

• Researchers must be aware of inherent biases when using current methods to infer power laws from rank-frequency data, especially in natural language.
• The assumption of simple probability distributions is a critical limitation for accurate Zipf exponent estimation in linguistics.
• Further development of methods accounting for the complexity of natural language processes is needed for unbiased power law inference.