A bilingual benchmark for evaluating large language models

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Computational Linguistics

Background:

• Large language models (LLMs) have advanced rapidly, yet their evaluation in Arabic is limited.
• Existing benchmarks do not facilitate direct bilingual comparison of LLM performance.
• Assessing LLM capabilities in Arabic is crucial for equitable technological development.

Purpose of the Study:

• To introduce a novel benchmark for the bilingual evaluation of LLMs in English and Arabic.
• To enable direct comparison of LLM performance across these two languages.
• To quantify the linguistic capabilities of LLMs, specifically ChatGPT and GPT-4, in both English and Arabic.

Main Methods:

• Development of a new evaluation dataset based on the General Aptitude Test (GAT).
• Conducting experiments to assess ChatGPT's performance in English versus Arabic.
• Investigating the impact of language switching in task descriptions.
• Comparing LLM performance with fastText for Arabic word analogies.

Main Results:

• ChatGPT demonstrates superior performance in English compared to Arabic.
• fastText outperforms ChatGPT in identifying Arabic word analogies.
• GPT-4 exhibits substantially enhanced Arabic linguistic capabilities compared to ChatGPT.
• GPT-4's Arabic performance approaches its English performance level.

Conclusions:

• The proposed benchmark effectively facilitates bilingual LLM evaluation.
• GPT-4 represents a significant advancement in Arabic language understanding for LLMs.
• Further research is needed to fully explore and enhance LLM capabilities in low-resource languages.