Establishing vocabulary tests as a benchmark for evaluating large language models

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Artificial Intelligence

Background:

• Traditional vocabulary tests were vital for language modeling evaluation but are now overlooked.
• Current Large Language Model (LLM) benchmarks often neglect fundamental linguistic aspects, focusing instead on task-specific or domain knowledge.
• There is a need to re-evaluate LLMs using foundational linguistic assessments.

Purpose of the Study:

• To advocate for the revival of vocabulary tests as a key method for assessing LLM performance.
• To investigate the lexical knowledge of prominent LLMs using standardized vocabulary tests.
• To identify and analyze gaps in LLM word representations and learning mechanisms.

Main Methods:

• Evaluation of seven distinct LLMs, including models like Llama 2, Mistral, and GPT.
• Utilized two different formats of vocabulary tests.
• Conducted assessments across two languages to ensure cross-linguistic validity.

Main Results:

• Identified surprising and significant gaps in the lexical knowledge of the evaluated LLMs.
• Observed performance variations across different LLM architectures and languages.
• Provided insights into the intricacies of LLM word representations and their underlying learning processes.

Conclusions:

• Vocabulary tests remain a valuable and necessary tool for comprehensive LLM evaluation.
• The findings highlight the limitations of current LLM evaluation practices and the need for a more holistic approach.
• Automated vocabulary test generation presents a scalable method for continuous LLM language skill assessment.