Are Some Words Worth More than Others?

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Artificial Intelligence

Background:

• Current language model evaluation metrics primarily focus on token-level accuracy against ground truth.
• Traditional metrics like perplexity can be skewed by the Zipfian distribution of language, overemphasizing frequent words.
• Existing methods overlook the linguistic utility of potentially mis-predicted words and model performance variations across word frequencies.

Purpose of the Study:

• To introduce novel intrinsic evaluation measures for language models.
• To provide a more holistic assessment of language model performance beyond simple accuracy.
• To address limitations of current metrics in capturing model behavior with varying word frequencies.

Main Methods:

• Development of two new intrinsic evaluation metrics for a word prediction task.
• Evaluation of several large English language models using the proposed metrics.
• Comparison of results from new metrics against traditional evaluation methods.

Main Results:

• The proposed metrics reveal functional performance differences between language models.
• These differences are not apparent when using traditional evaluation metrics.
• The new approach highlights model variations in handling high- and low-frequency words.

Conclusions:

• The novel evaluation measures offer a more nuanced understanding of language model capabilities.
• These metrics can identify potential failure modes, such as repetitive text generation.
• The findings suggest a need for more sophisticated evaluation techniques in natural language processing.