Distributed output formation tracking control of heterogeneous multi-agent systems using reinforcement learning

This paper studies the distributed time-varying output formation tracking problem for heterogeneous multi-agent systems with both diverse dimensions and parameters. The output of each follower is supposed to track that of the virtual leader while accomplishing a time-varying formation configuration. First, a distributed trajectory generator is proposed based on neighboring interactions to reconstitute the state of virtual leader and provide expected trajectories with the formation incorporated. Second, an optimal tracking controller is designed by the model-free reinforcement learning technique using online off-policy data instead of requiring any knowledge of the followers' dynamics. Stabilities of the learning process and resulting controller are analyzed while solutions to the output regulator equations are equivalently obtained. Third, a compensational input is designed for each follower based on previous learning results and a derived feasibility condition. It is proved that the output formation tracking error converges to zero asymptotically with the biases to cost functions being restricted arbitrarily small. Finally, numerical simulations verify the proposed learning and control scheme.