Exploring Dimensionality Reduction Techniques in Multilingual Transformers.

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Computational Linguistics

Background:

• Semantic and context-aware NLP solutions are crucial for complex Human Language Understanding tasks.
• Multilingual models are essential to overcome language barriers in NLP.
• Increasing model complexity necessitates efficient handling of high-dimensional embeddings.

Purpose of the Study:

• To comprehensively analyze the impact of various dimensionality reduction techniques on multilingual siamese transformers.
• To evaluate unsupervised dimensionality reduction methods including feature extraction, selection, and manifold techniques.
• To assess the performance of these techniques on semantic textual similarity tasks.

Main Methods:

• Applied diverse dimensionality reduction techniques (linear, nonlinear, feature selection, manifold) to multilingual siamese transformers.
• Utilized the multilingual extended Semantic Textual Similarity Benchmark (mSTSb) for evaluation.
• Compared embeddings from pre-trained models with fine-tuned STS versions.

Main Results:

• Achieved significant average dimensionality reduction in embeddings from pre-trained models.
• Demonstrated fitting times for dimensionality reduction that are faster than fine-tuning.
• Obtained substantial dimensionality reduction for embeddings from fine-tuned models.

Conclusions:

• Dimensionality reduction techniques can effectively reduce the size of high-dimensional embeddings for NLP tasks.
• These techniques offer a more efficient alternative to fine-tuning for certain applications.
• Findings contribute to understanding the interplay between tuning, dimensionality reduction, and performance in semantic-aware NLP.