Zipf's Law for Word Frequencies: Word Forms versus Lemmas in Long Texts

Area of Science:

• Computational Linguistics
• Natural Language Processing
• Quantitative Linguistics

Background:

• Zipf's law is a foundational principle in language statistics, describing word frequency distributions.
• Understanding the elementary units for Zipf's law is crucial for linguistic analysis.
• Morphological complexity influences language structure and word form variations.

Purpose of the Study:

• To investigate the validity of Zipf's law for both plain word forms and lemma forms.
• To analyze how the word-to-lemma transformation affects Zipf's law parameters (exponent and cut-off).
• To compare Zipf's law adherence across four languages with varying morphological complexity.

Main Methods:

• Analysis of several long literary texts in four different languages.
• Statistical examination of word form frequencies and corresponding lemma frequencies.
• Comparison of Zipf's law exponent and low-frequency cut-off before and after word-to-lemma transformation.

Main Results:

• Zipf's law is consistently fulfilled for both word and lemma forms across all analyzed texts.
• The exponents of Zipf's law show remarkable similarity between word and lemma forms, despite significant frequency range alterations.
• Low-frequency cut-offs demonstrate less stability, tending to increase substantially after the word-to-lemma transformation.

Conclusions:

• Zipf's law is a robust phenomenon applicable to both word and lemma levels in natural language.
• The word-to-lemma transformation preserves the general power-law distribution but alters specific parameters.
• Linguistic analysis should consider the impact of morphological normalization on frequency distribution parameters.