Unsupervised multi-sense language models for natural language processing tasks.

Area of Science:

• Natural Language Processing
• Machine Learning
• Computational Linguistics

Background:

• Existing language models (LMs) use single representations for words, failing to capture nuances of polysemous words.
• Lack of large-scale annotated data for word senses hinders development of multi-sense models.
• Current multi-sense models often operate within restricted contexts, ignoring word order and long-term dependencies.

Purpose of the Study:

• To propose a sense-aware framework capable of processing multi-sense word information without annotated data.
• To develop context representations that preserve word order and long-term dependencies.
• To introduce a new evaluation metric, unigram-normalized perplexity (PPLu), for multi-sense LMs.

Main Methods:

• A three-stage framework: context representation, unsupervised sense-labeling via clustering, and multi-sense LM (MSLM) learning.
• Development and theoretical verification of unigram-normalized perplexity (PPLu) for evaluating MSLMs across different vocabulary sizes.
• Implementation using both unidirectional and bidirectional architectures (LSTM, Transformers) within the MSLM framework.

Main Results:

• The proposed MSLM framework outperforms single-sense LMs (SSLMs) with identical architectures and parameters.
• The framework effectively processes multi-sense word information without requiring annotated datasets.
• Demonstrated superior performance on downstream tasks within the General Language Understanding Evaluation (GLUE) and SuperGLUE benchmarks.

Conclusions:

• The sense-aware framework provides a robust solution for handling polysemy in language models.
• The novel PPLu metric offers a reliable way to evaluate MSLMs, independent of vocabulary size.
• The developed MSLM shows significant improvements in language understanding and performance on complex NLP tasks.