Shapley Idioms: Analysing BERT Sentence Embeddings for General Idiom Token Identification

Area of Science:

• Computational Linguistics
• Artificial Intelligence
• Natural Language Processing

Background:

• Transformer-based language models, such as BERT, are pivotal in Natural Language Understanding (NLU).
• Idiom token classification is a complex task requiring diverse linguistic feature integration.
• Previous research explored topic, lexical, and syntactic features for idiom identification.

Purpose of the Study:

• To investigate the linguistic information encoded by BERT.
• To evaluate BERT's effectiveness in idiom token classification.
• To analyze the contribution of different features to BERT's performance.

Main Methods:

• Utilized BERT sentence embeddings for idiom token identification, comparing with Skip-Thought embeddings.
• Employed Shapley Values to rank idiomatic expressions for model training.
• Analyzed the types of linguistic information BERT prioritizes.

Main Results:

• BERT sentence embeddings outperformed Skip-Thought in idiom token identification.
• Idiom-intrinsic and topic-based properties were found crucial for model training.
• BERT demonstrated efficient encoding of topic, lexical, and syntactic information.

Conclusions:

• BERT exhibits robust natural language understanding capabilities.
• The model effectively encodes a wide range of linguistic information.
• BERT represents a significant advancement in NLU, despite ongoing semantic debates.