EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Recurrent Neural Networks (RNNs) excel at modeling temporal dependencies in sequential data.
• Existing RNN architectures primarily focus on managing temporal information flow.
• A gap exists in addressing the varying importance of individual elements within input vectors.

Purpose of the Study:

• To introduce a novel mechanism, the Element-wise-Attention Gate (EleAttG), to enhance RNN capabilities.
• To enable RNN neurons to selectively attend to different elements within an input vector.
• To improve the performance of RNNs on complex sequential data tasks.

Main Methods:

• Propose Element-wise-Attention Gate (EleAttG) as an add-on module for RNN blocks.
• EleAttG adaptively modulates input elements by assigning differential importance (attention).
• The enhanced RNN block is termed EleAtt-RNN, applicable to various RNN types (RNN, LSTM, GRU).

Main Results:

• EleAtt-RNN demonstrates effectiveness across diverse tasks: action recognition (skeleton and video), gesture recognition, and sequential MNIST classification.
• Element-wise, content-adaptive modulation of input significantly boosts RNN performance.
• The proposed EleAttG provides a fundamental, generalizable unit for improving RNNs.

Conclusions:

• The Element-wise-Attention Gate (EleAttG) offers a simple yet powerful method to enhance RNNs.
• Attentiveness at the element-wise level improves the modeling capacity of RNNs for sequential data.
• EleAtt-RNN represents a significant advancement in handling complex sequential information processing.