FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning

Reliable value estimation serves as the cornerstone of reinforcement learning (RL) by evaluating long-term returns and guiding policy improvement, significantly influencing the convergence speed and final performance. Existing works improve the reliability of value function estimation via multi-critic ensembles and distributional RL, yet the former merely combines multi point estimation without capturing distributional information, whereas the latter relies on discretization or quantile regression, limiting the expressiveness of complex value distributions. Inspired by flow matching's success in generative modeling, we propose a generative paradigm for value estimation, named FlowCritic. Departing from conventional regression for deterministic value prediction, Flow Critic leverages flow matching to model value distributions and generate samples for value estimation. Specifically, FlowCritic formulates the target value distribution through distributional Bellman operators, and trains a velocity field network to learn the continuous probability flow from a tractable prior distribution to the target distribution, thereby enabling flexible modeling of arbitrarily complex value distributions. More importantly, FlowCritic introduces the coefficient of variation (CoV) of the generated value distribution to quantify the noise level of training samples. BasedonCoV, FlowCritic assigns adaptive weights to prioritize low noise samples, thereby reducing the variance of policy gradient during backpropagation. Additionally, FlowCritic incorporates truncated sampling and velocity field clipping mechanisms to ensure training stability. Our theoretical analysis demonstrates FlowCritic's convergence and advantages. Extensive experiments on 12 IsaacGym benchmarks demonstrate FlowCritic's superiority over existing RL baselines, while its successful deployment on a real quadrupedal robot platform validates its effectiveness in practical physical systems. Notably, FlowCritic is the first approach to integrate flow matching into value distribution modeling for RL, and offers a new perspective on the effective exploitation of distributional information.