ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment.

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Machine Learning

Background:

• Existing multilingual readability assessment resources lack domain and language diversity.
• This limits cross-domain and cross-lingual analyses of language model performance.
• Developing robust methods requires diverse evaluation benchmarks.

Purpose of the Study:

• To introduce ReadMe++, a novel multilingual, multi-domain dataset for readability assessment.
• To benchmark large language models (LLMs) on multilingual readability.
• To encourage research in robust multilingual readability assessment.

Main Methods:

• Creation of ReadMe++: a dataset with 9757 human-annotated sentences across Arabic, English, French, Hindi, and Russian from 112 sources.
• Benchmarking LLMs using supervised, unsupervised, and few-shot prompting settings on ReadMe++.
• Evaluation of domain generalization and cross-lingual transfer capabilities.

Main Results:

• ReadMe++ enables testing of advanced few-shot prompting techniques.
• Identified limitations in current state-of-the-art unsupervised methods.
• Models trained on ReadMe++ demonstrated superior domain generalization.
• Enhanced cross-lingual transfer capabilities were observed in trained models.

Conclusions:

• ReadMe++ serves as a valuable benchmark for multilingual readability assessment.
• The dataset facilitates the development of more robust and generalizable LLMs.
• Public release of data and tools aims to advance the field.