subs2vec: Word embeddings from subtitles in 55 languages.

Area of Science:

• Computational Linguistics
• Psycholinguistics
• Natural Language Processing

Background:

• Word embeddings are crucial for representing lexical semantics numerically.
• Existing embeddings often rely on non-conversational text corpora like Wikipedia.
• A need exists for embeddings that capture the nuances of spoken language.

Purpose of the Study:

• To introduce a novel collection of word embeddings trained on pseudo-conversational speech.
• To evaluate the performance of these embeddings against traditional ones.
• To propose a new evaluation method relevant to psycholinguistics.

Main Methods:

• Training word embeddings using the fastText skipgram algorithm on the OpenSubtitles corpus.
• Utilizing pseudo-conversational speech transcriptions from television and movies.
• Evaluating performance on standard benchmark datasets and a novel lexical norm prediction task.

Main Results:

• The developed word embeddings demonstrate performance comparable to, and in some cases exceeding, those trained on Wikipedia.
• The novel evaluation method successfully predicts experimental lexical norms across multiple languages.
• The embeddings capture semantic representations from conversational data effectively.

Conclusions:

• Pseudo-conversational speech corpora are a viable source for training high-quality word embeddings.
• These new embeddings offer a valuable resource for computational linguistics and psycholinguistics.
• The freely available models and code facilitate further research and application.