Word2vec Skip-Gram Dimensionality Selection via Sequential Normalized Maximum Likelihood.

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Machine Learning

Background:

• Word embeddings like word2vec Skip-gram (SG) implicitly estimate word contextual distributions.
• Selecting the optimal dimensionality for SG models is crucial for accurate representation.
• Existing methods may not consistently identify the best dimensionality.

Purpose of the Study:

• To propose a novel information criteria-based method for selecting the optimal dimensionality of word2vec Skip-gram models.
• To evaluate the effectiveness of Akaike's Information Criterion (AIC), Bayesian Information Criterion (BIC), and Sequential Normalized Maximum Likelihood (SNML) for this task.
• To introduce efficient computation heuristics for SNML.

Main Methods:

• Applied information criteria (AIC, BIC, SNML) to determine SG model dimensionality.
• Considered SG and SG Negative Sampling models.
• Developed novel heuristics for efficient SNML computation.
• Evaluated dimensionality selection against word analogy and similarity tasks.

Main Results:

• Empirically demonstrated that SNML outperforms BIC and AIC in dimensionality selection.
• The dimensionality selected by SNML is closer to the optimal dimensionality determined by word analogy and similarity tasks.
• Proposed heuristics significantly improve SNML's computational efficiency.

Conclusions:

• Information criteria, particularly SNML, provide a robust method for selecting word2vec Skip-gram dimensionality.
• SNML offers superior performance compared to AIC and BIC for this task.
• The proposed approach enhances the quality of word embeddings by optimizing dimensionality.