Pre-Training a Graph Recurrent Network for Text Understanding.

Area of Science:

• Natural Language Processing
• Deep Learning Architectures
• Computational Linguistics

Background:

• Transformer models dominate Natural Language Processing (NLP) but suffer from quadratic time complexity and limited sentence-level expressiveness due to reliance on special tokens.
• Existing alternatives like CNNs, MLPs, and SSMs have been explored, but limitations in efficiency and representation quality persist.

Purpose of the Study:

• To propose a novel graph recurrent network (GRN) for language model pre-training that addresses the limitations of Transformer architectures.
• To achieve comparable performance to existing models with enhanced inference efficiency and improved representation quality.

Main Methods:

• Developed a graph recurrent network with linear time complexity for pre-training language models.
• Constructed a graph structure for each sequence, enabling local token-level communications.
• Introduced a detached sentence-level representation independent of normal tokens.

Main Results:

• The proposed GRN achieved comparable performance to Transformer-based models on English and Chinese text understanding tasks.
• Demonstrated significantly higher inference efficiency compared to existing pre-trained models.
• Discovered that the GRN generates more diverse and uniform representations, mitigating issues like representation degradation.

Conclusions:

• The graph recurrent network offers a viable and efficient alternative to Transformer models for language model pre-training.
• The GRN's architecture effectively addresses computational cost and representation limitations, leading to improved performance and efficiency.
• The model's ability to generate diverse and uniform representations enhances its applicability in various NLP tasks.