Zipf's law holds for phrases, not words

Area of Science:

• Computational Linguistics
• Statistical Physics
• Natural Language Processing

Background:

• Zipf's law, a principle of word frequency distribution, has limited applicability in linguistic analysis, typically failing after a few orders of magnitude.
• Existing models struggle to capture the full scaling behavior of language units.

Purpose of the Study:

• To investigate the applicability of Zipf's law to phrases as units of meaning in human language.
• To develop a novel statistical method for analyzing text based on phrase frequency and scaling.

Main Methods:

• Empirical analysis of phrase frequency distributions across a large corpus.
• Development of a statistical mechanical model for random text partitioning.
• Rank ordering of mixed-length phrases to test scaling properties.

Main Results:

• Zipf's law for phrases demonstrates remarkable scalability, extending over nine orders of rank magnitude.
• The proposed random text partitioning method provides a principled and scalable approach to text analysis.
• This finding challenges the limitations of Zipf's law in traditional word-based analyses.

Conclusions:

• Phrases, not just words, exhibit robust scaling behavior consistent with Zipf's law.
• The developed methodology enables a more comprehensive and rigorous analysis of textual data.
• This research opens new avenues for understanding language structure and meaning through scalable phrase analysis.