Burst and Memory-aware Transformer: capturing temporal heterogeneity.

Area of Science:

• Computational Neuroscience
• Machine Learning
• Time Series Analysis

Background:

• Event sequences exhibit temporal heterogeneity, appearing as burst patterns across diverse domains like neuronal firing and human activities.
• Existing Transformer models, while adept at capturing long-term dependencies via self-attention, struggle with the complexities of bursty temporal data.

Purpose of the Study:

• To introduce a novel Burst and Memory-aware Transformer (BMT) model designed to explicitly address and model temporal heterogeneity in event sequences.
• To enhance the predictive accuracy of event sequence models by incorporating burstiness and memory dynamics.

Main Methods:

• Developed the BMT model, integrating burstiness and memory coefficients directly into the self-attention mechanism of the Transformer.
• Introduced a novel loss function to jointly optimize burstiness, memory coefficients, and their discretized representations.
• Evaluated the model on synthetic and real-world datasets exhibiting power-law inter-event time distributions.

Main Results:

• The BMT model demonstrated superior performance in predicting event times and intensity functions compared to existing models and control groups.
• The model showed remarkable effectiveness on temporally heterogeneous data, particularly those with power-law inter-event time distributions.
• Incorporating burst-related parameters significantly improved the Transformer's ability to interpret heterogeneous event sequences.

Conclusions:

• The BMT model offers a significant advancement in predicting event sequences with complex temporal heterogeneity.
• Explicitly modeling burstiness and memory enhances the Transformer's comprehension of non-uniform event dynamics.
• The findings highlight the importance of burst-aware parameters for improving predictive performance in diverse real-world applications.