Sample Efficient Deep Reinforcement Learning With Online State Abstraction and Causal Transformer Model Prediction.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Deep reinforcement learning (RL) demands extensive training data, limiting practical applications.
• State abstraction and world models offer potential for enhanced sample efficiency but can degrade performance.

Purpose of the Study:

• To introduce a novel algorithm, Abstracted Model-based Policy Learning (AMPL), to significantly improve sample efficiency in deep RL.
• To address the performance degradation associated with traditional state abstraction and world model approaches.

Main Methods:

• Developed a state abstraction method using multistep bisimulation to create task-related latent state spaces, compressing Markov decision processes (MDPs) into abstracted MDPs.
• Designed a causal transformer model predictor (CTMP) to approximate abstracted MDPs and generate long-horizon simulated trajectories with reduced prediction error.
• Employed a modified multistep soft actor-critic algorithm with a λ-target for efficient policy learning within abstracted MDPs.

Main Results:

• Theoretical analysis confirms AMPL's ability to enhance sample efficiency during training.
• AMPL demonstrated superior sample efficiency compared to state-of-the-art deep RL algorithms on Atari games and the DeepMind Control (DMControl) suite.
• Empirical results on DMControl tasks with moving noises show AMPL's robustness to task-irrelevant distractors.

Conclusions:

• The proposed AMPL algorithm effectively improves sample efficiency in deep RL.
• AMPL offers a robust and high-performing solution for complex RL tasks, even in the presence of observational noise.