A²P-MANN: Adaptive Attention Inference Hops Pruned Memory-Augmented Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Memory-augmented neural networks (MANN) are powerful tools for complex reasoning tasks.
• However, MANNs often require a high number of attention inference hops, leading to significant computational overhead.
• Efficiently managing these computational resources is crucial for practical applications.

Purpose of the Study:

• To propose an online adaptive approach, termed [Formula: see text]-memory-augmented neural network ([Formula: see text]-MANN), to reduce attention inference hops in MANNs.
• To introduce weight pruning techniques for the fully connected layers of [Formula: see text]-MANN to further decrease computational costs.
• To evaluate the effectiveness of the proposed methods on question answering (QA) datasets and MANN architectures.

Main Methods:

• An online adaptive strategy using a small neural network classifier to determine the optimal number of attention inference hops for each input query.
• Development of two weight pruning approaches for the final fully connected layers: one with negligible accuracy loss and another with controllable accuracy trade-offs.
• Application and assessment of the [Formula: see text]-MANN approach on two distinct MANN structures and two QA datasets.

Main Results:

• Achieved an average reduction of 50% in computations compared to baseline MANNs, with less than 1% accuracy loss.
• Combined with the zero-skipping technique, the approach reduced computation counts by approximately 70%.
• Demonstrated an average runtime reduction of 43% on CPU and GPU platforms.

Conclusions:

• The proposed [Formula: see text]-MANN approach effectively reduces computational complexity in memory-augmented neural networks.
• Weight pruning further enhances efficiency with minimal impact on accuracy.
• The method offers significant performance improvements for QA systems and other applications leveraging MANNs.