GATmath and GATLc: Comprehensive benchmarks for evaluating Arabic large language models.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computational Linguistics

Background:

• Large Language Models (LLMs) have advanced AI, but their development requires robust evaluation.
• Assessing Arabic LLMs is hindered by a lack of comprehensive benchmarks and evaluation tools.
• This scarcity limits the progress and real-world application of Arabic language models.

Purpose of the Study:

• Introduce GATmath (7k questions) and GATLc (9k questions), novel Arabic benchmarks for multitask reasoning and language understanding.
• Provide the first large-scale, comprehensive reasoning dataset specifically designed for the Arabic language.
• Facilitate rigorous evaluation and drive the advancement of Arabic LLMs.

Main Methods:

• Created two large-scale Arabic datasets, GATmath and GATLc, derived from the General Aptitude Test (GAT).
• Datasets encompass diverse categories requiring reasoning, semantic analysis, language comprehension, and mathematical problem-solving.
• Evaluated seven prominent LLMs on these newly developed benchmarks.

Main Results:

• The highest-performing LLM achieved only 66.9% (GATmath) and 64.3% (GATLc) accuracy.
• These results highlight the significant difficulty posed by the GATmath and GATLc datasets.
• Current state-of-the-art LLMs demonstrate substantial limitations in Arabic reasoning and language understanding.

Conclusions:

• The GATmath and GATLc datasets present a considerable challenge for existing Arabic LLMs.
• There is substantial room for improvement in developing more capable Arabic language models.
• These benchmarks are crucial for advancing research and development in Arabic AI.