Deep belief Markov models for POMDP inference.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Partially Observable Markov Decision Processes (POMDPs) are crucial for sequential decision-making under uncertainty.
• Existing inference methods for high-dimensional POMDPs often struggle with scalability and lack of ground truth state data.
• Deep learning offers potential for modeling complex, non-linear dynamics inherent in these problems.

Purpose of the Study:

• Introduce a novel deep learning architecture, the Deep Belief Markov Model (DBMM), for efficient POMDP inference.
• Develop a model-formulation agnostic approach to handle complex, high-dimensional, and partially observable environments.
• Enable robust belief inference using only observational data, overcoming limitations of exact computation and sampling methods.

Main Methods:

• Developed the Deep Belief Markov Model (DBMM), extending deep Markov models to the POMDP framework.
• Utilized variational inference methods for efficient belief inference directly from observation data.
• Leveraged neural networks to infer and simulate non-linear system dynamics, accommodating high dimensionality and mixed variable types.

Main Results:

• DBMMs demonstrated efficient, model-formulation agnostic inference capabilities in benchmark POMDP problems with discrete and continuous variables.
• Neural network parameters were updated efficiently based on data availability, allowing dynamic adaptation.
• An RL agent guided by DBMM beliefs significantly outperformed model-free baselines, achieving near-optimal performance in a downstream task.

Conclusions:

• DBMMs provide an effective solution for belief inference in complex POMDPs, overcoming scalability and data limitations of traditional methods.
• The architecture's ability to infer beliefs enables the derivation of effective POMDP solutions.
• DBMMs show significant practical utility, enhancing reinforcement learning agent performance in challenging decision-making scenarios.