Latent MeanFlow Policy Optimization for Offline Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Offline reinforcement learning (RL) seeks policies from static datasets.
• Generative models like diffusion and flow matching improve policy expressiveness but are computationally expensive and rely on weak Gaussian priors.

Purpose of the Study:

• To introduce Latent MeanFlow Policy Optimization (LaMPO), a generative policy framework for efficient offline RL.
• To address the computational costs and representational limitations of existing generative models in offline RL.

Main Methods:

• LaMPO formulates offline RL as latent generative policy optimization.
• It learns a behavior-conditioned latent distribution for an informative prior.
• A MeanFlow policy predicts average action velocity for one-step action generation, avoiding iterative sampling.

Main Results:

• LaMPO achieves superior performance and high efficiency across 71 tasks in OGBench, D4RL, and humanoid manipulation.
• It demonstrates a 19% average improvement over state-of-the-art methods.
• Achieved an 81% success rate in real-world robotic tasks.

Conclusions:

• LaMPO offers an effective and efficient generative policy optimization framework for offline RL.
• The method successfully overcomes the limitations of previous generative approaches, enabling high-fidelity action generation and policy improvement.