Smart PID generator for nonlinear systems: zero-shot reinforcement learning based on virtual environments

Proportional-integral-derivative (PID) controllers remain the cornerstone of industrial automation owing to their robustness and operational simplicity. However, tuning PID parameters for nonlinear processes presents persistent challenges, often requiring system linearization and repeated recalibration under varying operating conditions. Although both traditional and reinforcement learning (RL)-based auto-tuning methods have shown considerable promise, their dependence on direct trial-and-error interactions with live processes raises substantial safety and feasibility concerns in industrial environments. This study presents a novel PID generator framework that leverages virtual environments to enable safe and efficient offline training. Simplified first-order plus dead-time systems are constructed to emulate the slow nonlinear dynamics of target processes, serving as interactive surrogate environments for RL agent training. To address performance discrepancies under varying operating conditions, a reward normalization mechanism-a critical yet previously underexplored component for reliable RL-based PID tuning-is proposed. Furthermore, an enhanced actor-critic architecture is adopted to further improve learning efficiency and policy convergence. Once training is complete, the agent autonomously generates optimal PID parameters for target processes without requiring online retraining or direct process interaction. The effectiveness and robustness of the proposed framework are validated through comprehensive numerical simulations and real-world industrial experiments. The results demonstrate its capacity to adaptively and reliably determine optimal PID parameters for nonlinear systems, thereby bridging the gap between advanced RL methodologies and practical industrial control applications.