Lagrangian-based safe Q-learning for nonlinear delayed stochastic systems

This paper addresses the safe optimal control problem for a class of continuous-time nonlinear Itô stochastic systems with state delays and unknown dynamics. We propose a safe Q-learning algorithm that, for the first time, extends this framework from linear deterministic systems to the nonlinear stochastic domain. The simultaneous presence of stochastic perturbations and state delays poses substantial challenges to safety enforcement and optimal policy learning. To address these challenges, a barrier Lyapunov function (BLF) and a Lagrangian saddle-point formulation are employed to characterize the safety-constrained optimal control problem. An adaptive safety multiplier estimation mechanism is then extended to stochastic systems with state delays, yielding a state- and delay-dependent safety mechanism that dynamically adjusts the intervention intensity and thereby alleviates the conservatism associated with fixed-gain safety intervention. Furthermore, a structured safe Q-function approximator and a model-free integral learning law are developed to avoid online system identification, whose computational burden and approximation errors can be exacerbated by state delays and stochastic perturbations. A Lyapunov-Krasovskii-based analysis establishes stochastic safety by deriving a uniform bound on the expected BLF together with a probabilistic compact-subset guarantee. It is further shown that the closed-loop state and the Critic weight estimation error are uniformly ultimately bounded in the mean-square sense.