Learning Continuous-Time Dynamics With Attention.

Area of Science:

• Machine Learning
• Dynamical Systems

Background:

• Sequential data analysis is vital, with attention mechanisms enhancing focus on relevant information.
• Traditional discrete-time attention struggles with irregularly sampled data, limiting its application.
• Continuous-time modeling offers a potential solution for handling data with non-uniform sampling intervals.

Purpose of the Study:

• To develop an attentive differential network (ADN) capable of learning from continuous-time dynamics in irregularly sampled sequence data.
• To address the limitations of traditional attention mechanisms in handling sparse or irregular time series.
• To propose causal (CADN) and latent CADN variants for improved efficiency and robustness.

Main Methods:

• Introduced an Attentive Differential Network (ADN) with continuous-time attention applied over dynamics at all time points.
• Developed a Causal Attentive Differential Network (CADN) by imposing a causality constraint to reduce memory costs.
• Explored a Latent CADN using an encoder-decoder structure with Bayesian learning for enhanced robustness.

Main Results:

• The proposed methods effectively compensate for missing information in irregular or sparse samples.
• Self-attention computation was optimized in CADN by restricting queries to the current time.
• Experiments demonstrated success in action recognition, emotion recognition, and mortality prediction using real-world irregularly sampled data.

Conclusions:

• Attentive Differential Networks provide a robust framework for analyzing continuous-time dynamics in irregularly sampled sequential data.
• CADN and latent CADN variants offer efficient and reliable solutions for complex sequence modeling tasks.
• The methods show significant promise across diverse applications, including bio-signal analysis and human behavior understanding.