Masked and Inverse Dynamics Modeling for Data-Efficient Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Deep reinforcement learning (DRL) faces challenges in data efficiency, particularly in learning state representations that evolve due to agent interaction.
• Existing methods integrating self-supervised learning (SSL) and data augmentation struggle to explicitly capture these changing state dynamics or select appropriate augmentations.

Purpose of the Study:

• To explicitly learn the inherent dynamics of changing states influenced by agent actions and environmental interactions.
• To improve data efficiency in pixel-based DRL by learning robust and agent-controllable state representations.

Main Methods:

• Proposed Masked and Inverse Dynamics Modeling (MIND), a self-supervised multitask learning framework using a transformer architecture.
• MIND employs masked modeling for static visual representations and inverse dynamics modeling for evolving state representations, utilizing masking augmentation.
• The method requires fewer hyperparameters and captures spatiotemporal information from consecutive frames.

Main Results:

• MIND demonstrated superior performance across discrete and continuous control benchmarks with limited interactions.
• The approach significantly improved data efficiency compared to previous methods.
• Successfully learned agent-controllable representations in dynamic environments.

Conclusions:

• MIND effectively learns evolving state representations by combining masked and inverse dynamics modeling.
• The proposed method offers a more data-efficient and robust approach to DRL in complex environments.
• The framework provides a promising direction for advancing DRL research and applications.