Unsupervised Sentence Representation Learning with Frequency-induced Adversarial tuning and Incomplete sentence...

Area of Science:

• Natural Language Processing
• Machine Learning

Background:

• Pre-trained Language Models (PLMs) are foundational for Unsupervised Sentence Representation Learning (USRL).
• PLMs exhibit anisotropic embedding spaces due to word frequency sensitivity, causing similarity and information bias, which degrades sentence embedding quality.

Purpose of the Study:

• To address the limitations of PLMs in USRL.
• To propose a novel framework, Slt-fai, for improving sentence representation learning by mitigating frequency-induced biases.

Main Methods:

• Calculated word frequencies from PLM pre-training corpora and assigned frequency labels.
• Developed a similarity discriminator for adversarial tuning to create frequency-invariant embeddings.
• Introduced an incomplete sentence detection task with an information discriminator to emphasize informative low-frequency words.

Main Results:

• Achieved a uniformly frequency-invariant embedding space.
• Enhanced the emphasis on informative low-frequency words.
• Demonstrated Slt-fai's superiority over existing USRL baselines across various backbones and datasets.

Conclusions:

• Slt-fai effectively overcomes frequency-induced biases in PLMs for USRL.
• The framework is flexible, plug-and-play, and enhances sentence embedding quality.
• Slt-fai offers a significant improvement for unsupervised sentence representation learning.