Probing language identity encoded in pre-trained multilingual models: a typological view.

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Machine Learning

Background:

• Pre-trained multilingual models are crucial for cross-lingual tasks.
• Current research prioritizes transfer performance over preserved linguistic properties like language identity.
• Understanding how models encode language identity is vital for effective cross-lingual information processing.

Purpose of the Study:

• To investigate the language identity preservation capabilities of state-of-the-art multilingual models (mBERT, XLM, XLM-R).
• To analyze how language typology influences the models' ability to retain language identity.
• To explore variations across different languages, typological features, and internal model layers.

Main Methods:

• Evaluated mBERT, XLM, and XLM-R on their capacity to preserve language identity.
• Analyzed model performance concerning language typology, including morphological, lexical, word order, and syntactic features.
• Examined the stability of language identity preservation across the hidden layers of each model.

Main Results:

• The ranking of models in preserving language identity is mBERT > XLM-R > XLM.
• All models effectively capture morphological, lexical, word order, and syntactic features.
• Models struggle with preserving nominal and verbal features.
• XLM-R and XLM exhibit stable language identity preservation across layers, while mBERT shows significant fluctuations.

Conclusions:

• mBERT demonstrates superior language identity preservation compared to XLM-R and XLM.
• While models capture certain linguistic features well, nominal and verbal features pose challenges.
• The layer-wise stability of language identity preservation varies significantly among these models.
• Findings offer valuable insights for selecting and refining multilingual models for cross-lingual applications.