Distinguishing subsampled power laws from other heavy-tailed distributions

Area of Science:

• Statistical analysis
• Data science
• Complex systems

Background:

• Distinguishing power-law distributions from other heavy-tailed distributions (lognormal, stretched exponential) is crucial in many scientific fields.
• Subsampling effects, common in network and biological sciences, can further complicate this detection process.

Purpose of the Study:

• To evaluate the performance of two common methods (Clauset et al.'s maximum likelihood and Voitalov et al.'s extreme value) in identifying power-law distributions from subsampled data.
• To assess how well these methods distinguish subsampled power laws from lognormal and stretched exponential distributions under random subsampling.

Main Methods:

• The study employed a random subsampling method, focusing on frequency distributions of elements (e.g., species, network nodes).
• Two established statistical methods were tested: maximum likelihood estimation and extreme value theory.
• The performance was evaluated across various subsampling depths to assess generalization to original distributions.

Main Results:

• Power-law exponents can be estimated with reasonable accuracy from subsamples, but correct distribution classification is more difficult.
• The maximum likelihood method frequently misclassifies subsampled power laws, rejecting the true hypothesis.
• The extreme value method shows better performance with subsampled power laws but struggles to differentiate them from other heavy-tailed distributions, with limitations often stemming from original sample classification.

Conclusions:

• Subsampling complicates the accurate identification of power-law distributions, impacting the reliability of common detection methods.
• The extreme value method demonstrates resilience to subsampling effects for power-law detection compared to the maximum likelihood method.
• Further research is needed to improve classification accuracy, as current methods show limitations in distinguishing between heavy-tailed distributions, particularly with subsampled data.