On cross-lingual retrieval with multilingual text encoders

Area of Science:

• Natural Language Processing
• Information Retrieval
• Machine Learning

Background:

• Pretrained multilingual transformer models (e.g., mBERT, XLM) are the current standard for cross-lingual transfer.
• Cross-lingual word embedding spaces (CLWEs) were previously dominant but are now largely superseded.
• The effectiveness of these transformers for cross-lingual retrieval tasks remains an open question.

Purpose of the Study:

• To systematically evaluate state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval.
• To compare their performance against earlier CLWE-based models in unsupervised settings.
• To investigate the impact of supervised fine-tuning and domain adaptation on retrieval performance.

Main Methods:

• Benchmarking unsupervised ad-hoc sentence- and document-level cross-lingual information retrieval (CLIR).
• Evaluating multilingual encoders fine-tuned on English relevance data for zero-shot transfer.
• Introducing and testing localized relevance matching for document retrieval.
• Conducting in-domain contrastive fine-tuning for language transfer.

Main Results:

• Multilingual encoders do not significantly outperform CLWEs for unsupervised document-level CLIR.
• State-of-the-art performance in sentence-level retrieval is achieved with supervised specialization, not off-the-shelf models.
• Supervised re-ranking rarely improves performance in zero-shot cross-lingual and domain transfer scenarios.
• In-domain contrastive fine-tuning is crucial for improving ranking quality.

Conclusions:

• Vanilla multilingual transformers are insufficient for unsupervised cross-lingual document retrieval.
• Supervised fine-tuning and domain adaptation are critical for achieving high performance in cross-lingual retrieval tasks.
• Significant differences exist between cross-lingual retrieval and monolingual retrieval transfer, indicating potential "monolingual overfitting".