Efficient Memory-Enhanced Transformer for Long-Document Summarization in Low-Resource Regimes

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Machine Learning

Background:

• Current transformer models struggle with long document summarization due to computational and memory constraints.
• Detecting long-range dependencies in extensive texts remains a significant challenge for state-of-the-art solutions.

Purpose of the Study:

• To introduce Emma, a novel, efficient memory-enhanced transformer architecture.
• To enable transformer models to process and comprehend entire documents of theoretically infinite length.
• To reduce the computational and memory demands of long document summarization.

Main Methods:

• Emma segments lengthy inputs into multiple text fragments.
• The model stores and compares the current text chunk with previous ones.
• This approach allows for context comprehension across the entire document using fixed GPU memory.

Main Results:

• Emma achieves competitive results on diverse datasets.
• The model significantly reduces GPU memory consumption during training and inference (18 GB and 13 GB, respectively).
• Emma demonstrates effectiveness even in low-resource settings.

Conclusions:

• Emma offers an efficient solution for long document summarization, overcoming current limitations.
• The architecture enables processing of theoretically infinite-length documents with manageable memory requirements.
• Emma provides a viable alternative for resource-constrained environments requiring advanced NLP capabilities.