Efficient Relational Sentence Ordering Network.

Area of Science:

• Natural Language Processing
• Deep Learning Architectures

Background:

• Effective modeling of coherence and inter-sentence relationships is crucial for various Natural Language Processing (NLP) tasks.
• Existing methods often struggle with computational efficiency and capturing complex relational information between sentences.

Purpose of the Study:

• To propose a novel deep Efficient Relational Sentence Ordering Network (ERSON) for improved coherence modeling.
• To enhance sentence ordering and related NLP tasks through efficient and effective relational information capture.

Main Methods:

• Introduction of a divide-and-fuse BERT (DF-BERT) model, refactoring BERT's encoder to independently encode sentences and jointly learn cross-attention between pairs.
• Development of a Relational Pointer Decoder (RPD) that leverages BERT's Next Sentence Prediction (NSP) task for capturing relative sentence order.
• Integration of knowledge distillation-based losses as auxiliary supervision to boost ordering performance.

Main Results:

• ERSON demonstrates superior performance on Sentence Ordering, Order Discrimination, and Multi-Document Summarization tasks.
• DF-BERT significantly reduces runtime and memory consumption without compromising model performance.
• The RPD effectively captures relative ordering information, enhancing prediction accuracy.

Conclusions:

• ERSON represents a significant advancement in sentence ordering and coherence modeling.
• The proposed DF-BERT and RPD components offer an efficient and effective approach to capturing inter-sentence relationships.
• ERSON outperforms state-of-the-art methods across multiple NLP benchmarks.