Learning Scalable Deep Kernels with Recurrent Structure.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Probabilistic Modeling

Background:

• Sequential data is prevalent in diverse fields like speech, robotics, finance, and biology.
• Standard kernel functions struggle to capture the inherent recurrent structures in sequential data.
• Long Short-Term Memory (LSTM) networks effectively model sequential data but lack probabilistic interpretability.

Purpose of the Study:

• To develop expressive, closed-form kernel functions for Gaussian processes (GPs) capable of modeling sequential data structures.
• To integrate the inductive biases of LSTMs with the non-parametric probabilistic advantages of GPs.
• To create a practical framework for Bayesian LSTMs.

Main Methods:

• Proposed novel, closed-form kernel functions tailored for Gaussian Process Long Short-Term Memory (GP-LSTM) models.
• Employed a provably convergent semi-stochastic gradient procedure to optimize GP marginal likelihood for kernel property learning.
• Leveraged kernel structure for scalable training and prediction in the GP-LSTM framework.

Main Results:

• Achieved state-of-the-art performance across multiple benchmark datasets.
• Demonstrated the practical utility of GP-LSTM in a critical autonomous driving application.
• Highlighted the unique value of predictive uncertainties offered by the GP-LSTM model.

Conclusions:

• GP-LSTM successfully integrates the strengths of Gaussian processes and LSTMs for robust sequential data modeling.
• The proposed method offers a practical and scalable approach to Bayesian recurrent neural networks.
• The model's ability to provide predictive uncertainties is particularly valuable for high-stakes applications like autonomous driving.