Emergent linguistic structure in artificial neural networks trained by self-supervision

Area of Science:

• Computational Linguistics
• Artificial Intelligence
• Cognitive Science

Background:

• Human language understanding relies on complex hierarchical structures not explicitly present in word sequences.
• Traditional approaches to modeling linguistic structure often require supervised learning on hand-labeled data.
• The mechanisms by which artificial neural networks acquire linguistic knowledge remain an active area of research.

Purpose of the Study:

• To investigate the emergent linguistic structural knowledge within large artificial neural networks trained via self-supervision.
• To develop methods for identifying and analyzing hierarchical linguistic structures learned by these models.
• To understand the contribution of learned linguistic structure to the performance of language models.

Main Methods:

• Utilizing self-supervised learning, where models predict masked words within a given context.
• Developing novel techniques to probe and visualize the internal representations of artificial neural networks.
• Analyzing the correlation between model embeddings and linguistic properties like parse tree distances.

Main Results:

• Demonstrating that deep contextual language models acquire significant aspects of syntactic grammatical relationships and anaphoric coreference without explicit supervision.
• Showing that linear transformations of learned embeddings can approximate parse tree distances.
• Successfully reconstructing approximate sentence tree structures from model outputs.

Conclusions:

• Self-supervised learning in large artificial neural networks enables the emergence of sophisticated linguistic structural knowledge.
• These findings provide insights into the internal workings of language models and their capabilities in natural language understanding.
• The learned structural representations contribute to the high performance of these models across various language tasks.