Seeds: Sampling-Enhanced Embeddings.

Area of Science:

• Natural Language Processing
• Machine Learning
• Computational Linguistics

Background:

• Static word embedding models like continue-bag-of-words (CBOW) and skip gram (SG) are cost-effective and interpretable alternatives to large pretrained models.
• Despite advances in dynamic models (e.g., BERT), static models remain crucial for many downstream applications.
• Negative sampling (NS) is a key component in sampling-based static word embedding models, but its core has critical problems.

Purpose of the Study:

• To investigate and mitigate critical problems within the negative sampling (NS) core of static word embedding models.
• To propose a novel algorithmic innovation for learning static word embeddings that enhances the sampling estimator.
• To introduce the Seeds framework, which dynamically considers multifactor global priors for different training pairs.

Main Methods:

• Developed the Seeds framework, a sampling-enhanced embedding approach.
• Implemented four concrete models: CBOW-GP, SG-GP (sampling negative words and positive auxiliaries), and CBOW-GN, SG-GN (sampling only negative instances).
• Evaluated model performance on a variety of standard intrinsic and extrinsic natural language processing tasks.

Main Results:

• Embeddings learned by the proposed Seeds models (CBOW-GP, SG-GP, CBOW-GN, SG-GN) consistently outperformed their negative sampling (NS) based counterparts (CBOW-NS, SG-NS).
• The new models also showed superiority over other strong baseline methods.
• The proposed methods demonstrated effectiveness across diverse intrinsic and extrinsic evaluation tasks.

Conclusions:

• The Seeds framework offers a significant improvement over traditional negative sampling estimators for static word embeddings.
• The proposed models provide a more effective and efficient approach to learning static word representations.
• Further research into sampling-based static word embedding models is warranted due to their continued relevance.