Word Embeddings as Statistical Estimators.

Area of Science:

• Natural Language Processing
• Statistical Theory
• Machine Learning

Background:

• Word embeddings are crucial in NLP but lack theoretical understanding.
• Current evaluation relies on empirical performance, not rigorous properties.
• Formal inference and uncertainty quantification require a theoretical basis.

Purpose of the Study:

• To provide a statistical theoretical perspective on word embeddings.
• To interpret classical methods like Word2Vec within a formal statistical model.
• To develop a novel, statistically tractable alternative to existing word embedding techniques.

Main Methods:

• Proposed a copula-based statistical model for text data.
• Interpreted Word2Vec as an estimator for theoretical pointwise mutual information (PMI).
• Developed a missing value-based estimator, building on prior work.

Main Results:

• Demonstrated Word2Vec's connection to estimating theoretical PMI.
• The proposed missing value estimator shows comparable estimation error to Word2Vec.
• The new estimator outperforms truncation-based methods.
• Achieved comparable performance to Word2Vec on an IMDb sentiment analysis task.

Conclusions:

• The copula-based model offers a theoretical foundation for word embeddings.
• The missing value estimator provides a statistically interpretable and effective alternative.
• This work bridges the gap between empirical success and theoretical understanding in word embeddings.